Search Results for "webcrawler"

crawler4j is an open source web crawler for Java which provides a simple interface for crawling the Web. Using it, you can setup a multi-threaded web crawler in few minutes. You need to create a crawler class that extends WebCrawler. This class decides which URLs should be crawled and handles the downloaded page. shouldVisit function decides whether the given URL should be crawled or not. In the above example, this example is not allowing .css, .js and media files and only allows pages within ics domain. visit function is called after the content of a URL is downloaded successfully. ...

AMPdoc mblazquez edition is a portable package with Apache, MySQL, PHP, and Perl which includes a selection of documentary software applications for libraries, archives, museums, publishers, conferences and documentation centers. AMPdoc mblazquez edition enables to perform workflow management duties, publication of contents, web positioning, among others. Usually, difficulty in installing and configuring of necessary applications has limited the access to technologies to information...

This is a simple webcrawler for FaceBook (TM) written in Java. The crawler will surf the public user pages (this means that you do not need to provide ann account) to reconstruct the friendship graph for further studies and analises

Spidertron is a multithreaded web crawling API for web sites of moderate size (hundreds of thousands of pages) that allows you to focus not on the crawling but on processing of the information retreived.

vinex (virtual network explorer) reduces the complexity of social networks by visualizing their structures. Based on data collected by a webcrawler from virtual communities, vinex offers the possibility to explore relationships between community members.