Search Results for "web crawler bot"

WebMagic is a scalable crawler framework. It covers the whole lifecycle of crawler, downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler. WebMagic is a simple but scalable crawler framework. You can develop a crawler easily based on it. WebMagic has a simple core with high flexibility, a simple API for html extracting. It also provides annotation with POJO to customize a crawler, and no configuration is needed. Some other...

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.

A Java Web crawler: multi-threaded, scalable, with high performance, extensible and polite. It can be used to crawl and index any web or enterprise domain and is configurable through a XML configuration file.

Web-as-corpus tools in Java. * Simple Crawler (and also integration with Nutch and Heritrix) * HTML cleaner to remove boiler plate code * Language recognition * Corpus builder

LogCrawler is an ANT task for automatic testing of web applications. Using a HTTP crawler it visits all pages of a website and checks the server logfiles for errors. Use it as a "smoketest" with your CI system like CruiseControl.

JIB - Java IM Bot, aimed to built a pluggable, customizable, expandable and dynamic bot framework for instant messager. With JIB user can control remote IM Bots over web based interface, configure bots to perform various tasks or custom behavior.

Spiritbot is a suite of tools designed for use in software-engineering, CodeGenUML includes an XMI/MOF/UML reader and exporter. An IRC ticker bot is also provided to aid in the development process.

IMShell is a specification and a set of utilities used to extend the command-line metaphor of to provide a command shell, which allows developers to easily add utilities to the environment. The environment itself will basically be an IM Bot.