Search Results for "crawl"

...Scraped pages can be streamed to a file, so if there's an error on one page, you won't lose what you've already scraped. Start on one page and move to the next easily. The flow is predictable, following a breadth-first crawl through each of the pages. X-ray has support for concurrency, throttles, delays, timeouts and limits to help you scrape any page responsibly. Swap in different scrapers depending on your needs. Currently supports HTTP and PhantomJS driver drivers. In the future, I'd like to see a Tor driver for requesting pages through the Tor network.

A Site Search engine script that uses MySQL to store your website's indexed pages, to add Search Functionality to Your Web Site. It is build with PHP and JavaScript, the search results are loaded via Ajax. The search system combine MySQL full text with SQL regexp, and words weight according to their location in the HTML elements, to determine the relevance of the search results. It can be included in any web site.

With PubSearch you can search for publications of an author in more publication databases at one time. PubSearch can crawl also "cited by" publications transitively for you! You can export publications to citation formats. You can add your own format templates and publication database definitions. See the link below for more details.

...Sites that publicize their access logs, including referer statistics, will then inadvertently link back to the spammer's site. These links will be indexed by search engines as they crawl the access logs. This benefits the spammer because of the free link, which gives the spammer's site improved search engine ranking due to link-counting algorithms that search engines use.

Law Leecher is a multi-threaded web crawling tool which extracts laws from the EU law database PreLex (http://ec.europa.eu/prelex/). It's written in Ruby.

nxs crawler is a program to crawl the internet. The program generates random ip numbers and attempts to connect to the hosts. If the host will answer, the result will be saved in a xml file. After than the crawler will disconnect... Additionally you can

Retriever is a simple crawler packed as a Java library that allows developers to collect and manipulate documents reachable by a variety of protocols (e.g. http, smb). You'll easily crawl documents shared in a LAN, on the Web, and many other sources.

Simple Porn Downloader is a tiny all Java based application that uses a list of keywords and starting urls to crawl webpages and branch out searching for specific media extensions which are downloaded and presented in an html page.

InfoCrawler allows you to crawl and index various types of documents, accessing data from various resources: Intranets, public WEB sites, local or remote file systems. For product information please see our website at http://www.infocrawler.org/

Sharehound is a network file systems indexer and searcher written in Java. Currently supports SMB file shares (i.e. MS Windows-based shares) and FTP resources. Web UI is used for search and crawl monitoring. RSS feed is provided for search results.

Ruya is a Python-based breadth-first, level-, delayed, event-based-crawler for crawling English, Japanese websites. It is targeted solely towards developers who want crawling functionality in their projects using API, and crawl control.

Crawl-By-Example runs a crawl, which classifies the processed pages by subjects and finds the best pages according to examples provided by the operator. Crawl-By-Example is a plugin to the Heritrix crawler, and was done as a part of GSoC06 program.

Toke is a webmining toolkit for web exploring, indexing and searching for Java. Toke allows to you crawl public or private web sites, in order to create web estatistics, web Pajek graphs, Lucene indexs and word frequency files for data clustering.

FTPSeek is an opensource (GPL) script which has the capability to crawl multiple FTP servers, allowing you to instantly search multiple servers at once, and browse through the file structures of each server at your will.

FemFind is a crawler/search engine for SMB shares (which can be found on Windows or Unix systems running Samba). FemFind does also crawl FTP servers and provides a web interface and a Windows client as frontends for searching.

Spindexer is a Search Engine/Crawler tool similar to UDMsearch or ht://dig - but unlike these tools, Spindexer is *very* fast and flexible. A simple Perl script works as a front-end to Pavuk and Swish++, allowing a fast crawl across any site(s).