SiteOne Crawler is a website analyzer and exporter
Goutte, a simple PHP Web Scraper
Library for Rapid (Web) Crawler and Scraper Development
Distributed web crawler admin platform for spiders management
Create and generate sitemaps with ease
The complete web scraping toolkit for PHP
Multiple server/webapp vulnerability scanner
An open source search engine with RESTFul API and crawlers
File system crawler and disk space usage software
IOSec Addons are enhancements for web security and crawler detection
Enumeration and automation of file discovery for your sec tools.