Crawler

Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page. More open source at https://github.com/fcc.

Project Activity

See All Activity >

License

GNU General Public License version 2.0 (GPLv2)

Follow Crawler

Crawler Web Site

Other Useful Business Software

Catch Bugs Before Your Customers Do

Real-time error alerts, performance insights, and anomaly detection across your full stack. Free 30-day trial.

Move from alert to fix before users notice. AppSignal monitors errors, performance bottlenecks, host health, and uptime—all from one dashboard. Instant notifications on deployments, anomaly triggers for memory spikes or error surges, and seamless log management. Works out of the box with Rails, Django, Express, Phoenix, Next.js, and dozens more. Starts at $23/month with no hidden fees.

Try AppSignal Free

Rate This Project

User Reviews

Be the first to post a review of Crawler!

Additional Project Details

Intended Audience

Government

User Interface

Web-based

Programming Language

PHP

Database Environment

SQL-based

Related Categories

PHP Search Engines, PHP Site Management Software

Registered

2010-09-03

Similar Business Software

ispmanager

Ispmanager – websites and web environment Linux control panel that makes life easier for web developers and hosting professionals. The main features: - Work and download a popular CMS: WordPress, Drupal, Joomla - Manage files through the built-in file manager - Choose from various web...

See Software
Perplexity

Where knowledge begins. Perplexity is an AI search engine that gives you quick answers. Available for free at as a web app, desktop app, or on the go on iPhone or Android. Perplexity AI is an advanced search and question-answering tool that leverages large language models to provide...

See Software
Google

Our mission is to organize the world's information so that everyone can access and use it. Every time you search, there are thousands, sometimes millions, of webpages with helpful information. How Google figures out which results to show starts long before you even type, and is guided by a...

See Software
Dekoo

Dekoo is the Asia's independent global search engine company offering Unique search, Unbiased results on the web having its own crawler, indexing, search and ranking algorithms along with AI/ML technologies with Tier 4 Datacenters and Content Delivery Network around the world. It aims to...

See Software
Plesk

The complete set of features to build, secure and run. Everything you'll need to effortlessly build your projects, secure against vulnerabilities and automate your day-to-day tasks - on one simple dashboard. Add domains and subdomains and move them between subscriptions. Configure DNS and...

See Software
Gigablast

Gigablast is a search engine designed to provide users with efficient and fast search results. It aggregates data from various sources, giving access to web pages, images, and more, allowing for a comprehensive search experience. Gigablast is particularly focused on simplicity and speed,...

See Software