AWBot is an easy to use tool to test a web site.:
AWbot connects to your web site and make URL requests like any other visitors.
AWBot is not a web indexing robot but a web client tool that emulate some visitors browsing on your site to test its reliab
FoxBAT was made in an attempt to see if Naïve Bayesian filtering commonly used for spam filtering could be employed in the World Wide Web context. The application consists of a Firefox extension (.XPI package) and a Perl server script.
Yngvi allows Apache Webmasters to determine filesystem locations that are accessible through the server -- including those which have not been explicitly published. It's good for finding inadvertent exposures or unintended configuration inheritances.
rss2imap is a tool which gets RSS (RDF Site Summary) from web and delivers to the IMAP server as an e-mail message. it enables you to use IMAP supported MUA as a RSS reader, and to unify mail check and site update check with ONE client.