Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page. More open source at https://github.com/fcc.
License
GNU General Public License version 2.0 (GPLv2)Follow Crawler
Other Useful Business Software
Get Avast Free Antivirus | Your top-rated shield against malware and online scams
Our antivirus software scans for security and performance issues and helps you to fix them instantly. It also protects you in real time by analyzing unknown files before they reach your desktop PC or laptop — all for free.
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of Crawler!