Search Results for "python web crawler"

crawler4j is an open source web crawler for Java which provides a simple interface for crawling the Web. Using it, you can setup a multi-threaded web crawler in few minutes. You need to create a crawler class that extends WebCrawler. This class decides which URLs should be crawled and handles the downloaded page. shouldVisit function decides whether the given URL should be crawled or not. In the above example, this example is not allowing .css, .js and media files and only allows pages within...

This is the Open Source RESTful client for the take.io platform.

The Java Sitemap Parser can parse a website's Sitemap (http://www.sitemaps.org/). This is useful for web crawlers that want to discover URLs from a website that is using the Sitemap Protocol. This project has been incorporated into crawler-commons (https://github.com/crawler-commons/crawler-commons) and is no longer being maintained.

Client libraries for the D-Wave Orion Web Service

This is a collection of REST specifications, and implementations of those specs, for very low-level information sharing and workflow operations using REST actions over HTTP. Implementations are in various languages, mainly Java, Python, and Ruby.

IMPORTANT : The project is now being hosted at http://code.google.com/p/ciform Plug'n Auth is an API providing easy integration of different authentication mechanisms into web applications. Within a few steps web admins will be able to change both the authentication backend and the logon frontend at any time with no further effort