Crawler

Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page. More open source at https://github.com/fcc.

Project Activity

See All Activity >

License

GNU General Public License version 2.0 (GPLv2)

Follow Crawler

Crawler Web Site

Other Useful Business Software

Our Free Plans just got better! | Auth0

With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.

Try free now

Rate This Project

User Reviews

Be the first to post a review of Crawler!

Additional Project Details

Intended Audience

Government

User Interface

Web-based

Programming Language

PHP

Database Environment

SQL-based

Related Categories

PHP Search Engines, PHP Site Management Software

Registered

2010-09-03

Similar Business Software

Dekoo

Dekoo is the Asia's independent global search engine company offering Unique search, Unbiased results on the web having its own crawler, indexing, search and ranking algorithms along with AI/ML technologies with Tier 4 Datacenters and Content Delivery Network around the world. It aims to...

See Software
Plesk

The complete set of features to build, secure and run. Everything you'll need to effortlessly build your projects, secure against vulnerabilities and automate your day-to-day tasks - on one simple dashboard. Add domains and subdomains and move them between subscriptions. Configure DNS and...

See Software
Gigablast

Gigablast is a search engine designed to provide users with efficient and fast search results. It aggregates data from various sources, giving access to web pages, images, and more, allowing for a comprehensive search experience. Gigablast is particularly focused on simplicity and speed,...

See Software
ispmanager

Ispmanager – websites and web environment Linux control panel that makes life easier for web developers and hosting professionals. The main features: - Work and download a popular CMS: WordPress, Drupal, Joomla - Manage files through the built-in file manager - Choose from various web...

See Software
Perplexity

Where knowledge begins. Perplexity is an AI search engine that gives you quick answers. Available for free at as a web app, desktop app, or on the go on iPhone or Android. Perplexity AI is an advanced search and question-answering tool that leverages large language models to provide...

See Software
Exa

The Exa API retrieves the best content on the web using embeddings-based search. Exa understands meaning, giving results search engines can’t. Exa uses a novel link prediction transformer to predict links which match the meaning of a prompt. For queries that need semantic understanding,...

See Software