Search Results for "extensible web spider"

Free website monitoring software, easy to set up and use for monitoring web sites. It is a web application programmed in Java programming language. You can monitor HTML pages, JSON and XML, pages in sitemap and even your whole web site using spider. Naturally you can check multiple websites. You can check HTTP result codes and even contents of the checked pages. Website checking is done periodically using build-in cron mechanism.

More than just a link checker, sitecheck is a website spider (also known as a crawler) which can assist with SEO by testing an entire site plus both inbound links from search engines and outbound links to other sites for the following issues: looping redirects (HTTP 301/302), broken links (HTTP 404), server errors (HTTP 500), spelling mistakes, low readability scores (using the Flesch Reading Ease test), missing/empty/duplicate meta tags, duplicate content, slow page speed, W3C validation...

Xaraya is an Open Source web application framework and content management solution written in PHP and licensed under the GNU General Public License. Xaraya is extensible, uses robust permissions, and multilingual systems to dynamically manage content. Track and get the latest developments on GitHub: https://github.com/xaraya

JLinkCheck is an Ant Task written in Java for checking links in websites. It is not just checking one single page, but crawling a whole site like a spider, generating a report in XML and (X)HTML. JReptator will be its succesor with many more features

Sperowider Website Archiving Suite is a set of Java applications, the primary purpose of which is to spider dynamic websites, and to create static distributable archives with a full text search index usable by an associated Java applet.

Robust featureful multi-threaded CLI web spider using apache commons httpclient v3.0 written in java. ASpider downloads any files matching your given mime-types from a website. Tries to reg.exp. match emails by default, logging all results using log4j.

A Java implementation of a flexible and extensible web spider engine. Optional modules allow functionality to be added (searching dead links, testing the performance and scalability of a site, creating a sitemap, etc ..

phpSitemapNG is a free Google Sitemaps generator written in PHP, but also generates RSS-based, txt-based and HTML-based sitemap files. It will spider your website and can also index the filesystem.