AI-Crawler Files

Crawl a website starting from a URL, find relevant pages

This is an exact mirror of the AI-Crawler project, hosted at https://github.com/oxylabs/ai-crawler-py. SourceForge is not affiliated with AI-Crawler.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
ai-crawler-py source code.tar.gz	2026-04-02	121.9 kB	0
ai-crawler-py source code.zip	2026-04-02	121.4 kB	2
Totals: 2 Items		243.4 kB	2

Other Useful Business Software

Our Free Plans just got better! | Auth0

With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.

Try free now

Compliant and Reliable File Transfers Backed by Top Security Certifications

Cerberus FTP Server delivers SOC 2 Type II certified security and FIPS 140-2 validated encryption.

Stop relying on non-certified, legacy file transfer tools that creak under the weight of modern security demands. Get full audit trails, advanced access controls and more supported by an award-winning team of experts. Start your free 25-day trial today.

Start Free Trial

Go From AI Idea to AI App Fast

One platform to build, fine-tune, and deploy ML models. No MLOps team required.

Access Gemini 3 and 200+ models. Build chatbots, agents, or custom models with built-in monitoring and scaling.

Try Free

Recommended Projects

GPT Crawler
Crawl a site to generate knowledge files to create your own custom GPT
Firecrawl
Turn entire websites into LLM-ready markdown or structured data
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
X-Crawl
Flexible Node.js AI-assisted crawler library
wombat
Lightweight Ruby DSL for scraping structured data from web pages

AI-Crawler Files

Crawl a website starting from a URL, find relevant pages

Get an email when there's a new version of AI-Crawler