Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page. More open source at https://github.com/fcc.
License
GNU General Public License version 2.0 (GPLv2)Follow Crawler
Other Useful Business Software
Stop vibe-debugging.
AppSignal's MCP server hands Claude, Cursor, or Zed your real errors, traces, and the deploy that shipped them. AI writes the fix; you review the diff.
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of Crawler!