If you are browsing any category, there is an error "Whoops, we can't find that page." if you go to any page past 100, making is difficult to find all projects.
That behavior is actually by design.
We currently limit category pagination to 100 pages, which is why you see the “Whoops, we can't find that page.” message beyond that point.
Please note that this URL includes various page types, so it will need to be filtered as needed (for example, to exclude comparison pages).
Could you let us know the intended purpose of crawling our listings and how the data will be used? Also, will you be using a specific User-Agent header when crawling? If so, please share the details.
Sincerely,
SourceForge Support
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
Thank you for pointing that out!
That behavior is actually by design.
We currently limit category pagination to 100 pages, which is why you see the “Whoops, we can't find that page.” message beyond that point.
If you'd like to find the full list of projects, you should use our sitemap files:
https://sourceforge.net/sitemap.xml
Sincerely,
SourceForge Support
Thanks! I notice this sitemap is just for the open source project files. Is there a different sitemap for the business software links?
Hi,
You can access all of our business listings here:
https://sourceforge.net/software_sitemap.xml
Please note that this URL includes various page types, so it will need to be filtered as needed (for example, to exclude comparison pages).
Could you let us know the intended purpose of crawling our listings and how the data will be used? Also, will you be using a specific User-Agent header when crawling? If so, please share the details.
Sincerely,
SourceForge Support
Hello,
We have not heard from you and are now closing this ticket. If you need any help moving forward, please submit another ticket. Thank you.
Sincerely,
SourceForge Support