SiteOne Crawler is a website analyzer and exporter
A scalable web crawler framework for Java
Goutte, a simple PHP Web Scraper
A free, feature-rich web analyzer and exporter/cloner you will love!
Internet Archive's open-source, web-scale, web crawler project
Distributed Crawler Management Framework Based on Scrapy
Create and generate sitemaps with ease
Dark Web OSINT Tool
Lighter, faster browser kernel of blink to integrate HTML UI in apps
The easiest way to add search to your documentation
Elasticsearch File System Crawler (FS Crawler)
Most Advanced Open Source Intelligence (OSINT) Framework
Python bindings for the Chromium Embedded Framework (CEF)
Elegant Scraper and Crawler Framework for Golang
Fast Middleware detect bot crawler for Koa
A website crawler and links downloader command line tool
Python ShadowSocks framework
Catberry is an isomorphic framework
BitTorrent DHT Protocol && DHT Spider.
Web Crawler/Spider for NodeJS + server-side jQuery
Fast Remote SVN Info
An awesome Go concurrent Crawler(spider) framework
Modular web site spider for web developers.
Zoozle 2008 - 2010 Webpage, Tools and SQL Files