Search Results for "heritrix"

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.

...Project goals: - Web page downloads for less experienced users, including easy setup - Project based page maintanance - Not too plain functions that include also multiple snapshots per project - Iterative, understandable and storage efficient data structure to enable more manual control over stored pages (meta files editable with Easy Folder Morpher) - Retain archived files and query links as original, altering links only during query Current status: - Alpha stadium, archivation quality below Heritrix...

...Throughout the project a large number of components have been developed to collect content from Web and Social Web, to analyse it from semantic and social perspectives and to enable Web archive access by different facets. The whole system based on the Heritrix crawler is released as open source to the public. Since many components or composite tools are of interest also for other areas and usage scenarios, the ARCOMEM consortium defined a number of pre-packaged tools which can be used independently from each other. By combining all packages the full ARCOMEM system can be build. The following major packages will be released in the coming weeks as pre-compiled packages with source code. ...

The archive-crawler project is building Heritrix: a flexible, extensible, robust, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accesible content.

Web-as-corpus tools in Java. * Simple Crawler (and also integration with Nutch and Heritrix) * HTML cleaner to remove boiler plate code * Language recognition * Corpus builder

The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.

Crawl-By-Example runs a crawl, which classifies the processed pages by subjects and finds the best pages according to examples provided by the operator. Crawl-By-Example is a plugin to the Heritrix crawler, and was done as a part of GSoC06 program.

Heritrix expand project