Search Results for "crawler"

This is a high-performance, production-tested library for parsing and evaluating robots.txt rules against crawler user agents. It implements the core semantics of the Robots Exclusion Protocol: user-agent sections, Allow/Disallow directives, wildcard handling, and precedence rules. The code is optimized for speed and low memory so large crawls can evaluate millions of URLs quickly. It also focuses on correctness—edge cases like overlapping patterns and longest-match resolution are handled consistently. ...

Visit our new site: http://socnetv.org Social Network Visualizer (SocNetV) is a social network analysis and visualization application. You can draw a social network (graph/digraph) or load an existing one (GraphML, UCINET, Pajek, etc), compute cohesion, centrality, community and structural equivalence metrics and apply various layout algorithms based on actor centrality or prestige scores (i.e. Eigenvector, Betweenness) or on dynamic models (i.e. Kamada-Kawai spring-embedder)

Universal information crawler is a fast precise and reliable Internet crawler. Uicrawler is a program/automated script which browses the World Wide Web in a methodical, automated manner and creates the index of documents that it accesses.

ICECrawler is a WWW crawler and map-generator intended to help understanding and analyzing links between websites and webdocuments.