Search Results for "website spider"

Free website monitoring software, easy to set up and use for monitoring web sites. It is a web application programmed in Java programming language. You can monitor HTML pages, JSON and XML, pages in sitemap and even your whole web site using spider. Naturally you can check multiple websites. You can check HTTP result codes and even contents of the checked pages. Website checking is done periodically using build-in cron mechanism. In case of a check failure, application will automatically...

More than just a link checker, sitecheck is a website spider (also known as a crawler) which can assist with SEO by testing an entire site plus both inbound links from search engines and outbound links to other sites for the following issues: looping redirects (HTTP 301/302), broken links (HTTP 404), server errors (HTTP 500), spelling mistakes, low readability scores (using the Flesch Reading Ease test), missing/empty/duplicate meta tags, duplicate content, slow page speed, W3C validation...

Sitemap Creator is a PHP class which creates XML sitemaps files compatible with the standard sitemaps.org protocol supported by Google and Bing. Features Uses PHPCrawl class to crawl/spider the website and creates URLs set while all PHPCrawl methods and options are accessible through class. Ability to calculate Priority, Frequency and Last-Modified date with variety of options. Creates sitemaps in gzip format or uncompressed XML. Pings search engines with sitemaps locations. Reads from...

SEO Tools provides tools for search engine optimization. Spider SEO: generates meta tags for all pages of a website. Web Page Optimizer performs analysis and changes in a web page. A graphical user interface is linked with a scriptol program that gene

This project will provide a tool for users to get a better understanding of the content and structure of an existing website. It will do this by providing a customised web spider as well as extensions to the GUESS graph visualisation application.

This CGI program can trap malicious robots that spider your website. The program works by blocking access from the bot's ip address. It can also provide an unlimited number of false e-mail addresses to muck up databases of email harvesting bots.

Sperowider Website Archiving Suite is a set of Java applications, the primary purpose of which is to spider dynamic websites, and to create static distributable archives with a full text search index usable by an associated Java applet.

Robust featureful multi-threaded CLI web spider using apache commons httpclient v3.0 written in java. ASpider downloads any files matching your given mime-types from a website. Tries to reg.exp. match emails by default, logging all results using log4j.

phpSitemapNG is a free Google Sitemaps generator written in PHP, but also generates RSS-based, txt-based and HTML-based sitemap files. It will spider your website and can also index the filesystem.