Spider Files

High-performance Rust web crawler and scraper for large-scale data

This is an exact mirror of the Spider project, hosted at https://github.com/spider-rs/spider. SourceForge is not affiliated with Spider.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
README.md	2026-03-20	1.3 kB	0
v2.47.75 source code.tar.gz	2026-03-20	776.1 kB	0
v2.47.75 source code.zip	2026-03-20	932.5 kB	0
Totals: 3 Items		1.7 MB	0

What's New

PageData & Crawler trait abstractions for extensible crawl pipelines
Proxy support for LLM HTTP requests (#378)
Chrome remote_addr via CDP Network.responseReceived
Remote cache for Chrome responses — dump & fallback support

Performance

SIMD-accelerated byte scanning (memchr), unrolled FNV hash
Trie: Box<str> keys + manual byte-walk + memchr dot scan
Bloom filter bitmask addressing + inline early-exit
Zero-alloc DNS cache hits via Arc<[SocketAddr]>
Skip robots.txt for single-page crawls, TCP keepalive always
Batch LRU cache eviction to reduce write-lock hold time
cache_mem auto-enables skip_browser for Chrome mode
Remove unnecessary Box allocations and redundant string conversions

Fixes

Fall back to HTTP crawl when Chrome is unavailable (#373)
Skip redirect-caused net::ERR_ABORTED instead of aborting navigation
Deterministic CDP event listener shutdown via watch channel
Remote cache fallback + 3s timeout for chrome_remote_cache path
Add decentralized stubs for set_url_parsed_direct and links_full

Deps

strum 0.28, phf 0.13, gemini-rust 1.7, chromey 2, llm_models_spider 0.1

Full Changelog: https://github.com/spider-rs/spider/compare/v2.47.52...v2.47.75

Source: README.md, updated 2026-03-20

Other Useful Business Software

MongoDB Atlas runs apps anywhere Icon

MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.

Start Free

Stop Storing Third-Party Tokens in Your Database Icon

Stop Storing Third-Party Tokens in Your Database

Auth0 Token Vault handles secure token storage, exchange, and refresh for external providers so you don't have to build it yourself.

Rolling your own OAuth token storage can be a security liability. Token Vault securely stores access and refresh tokens from federated providers and handles exchange and renewal automatically. Connected accounts, refresh exchange, and privileged worker flows included.

Try Auth0 for Free

MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.

Start Free

Recommended Projects

Abot
Fast and flexible C# framework for building customizable web crawlers
grab-site
Web crawler for archiving and backing up sites into WARC archives
Firecrawl
Turn entire websites into LLM-ready markdown or structured data
katana
Fast CLI web crawler for discovering endpoints in modern web apps
Dungeon Crawl Reference
Dungeon Crawl Stone Soup is a free rogue-like game of exploration and treasure-hunting. Stone Soup is a continuation of Linley's Dungeon Crawl. It is openly developed and invites participation from the Crawl community. See http://crawl.develz.org !