Crawler

Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page. More open source at https://github.com/fcc.

Project Activity

See All Activity >

License

GNU General Public License version 2.0 (GPLv2)

Follow Crawler

Crawler Web Site

Other Useful Business Software

Gen AI apps are built with MongoDB Atlas

Build gen AI apps with an all-in-one modern database: MongoDB Atlas

MongoDB Atlas provides built-in vector search and a flexible document model so developers can build, scale, and run gen AI apps without stitching together multiple databases. From LLM integration to semantic search, Atlas simplifies your AI architecture—and it’s free to get started.

Start Free

Rate This Project

User Reviews

Be the first to post a review of Crawler!

Additional Project Details

Intended Audience

Government

User Interface

Web-based

Programming Language

PHP

Database Environment

SQL-based

Related Categories

PHP Search Engines, PHP Site Management Software

Registered

2010-09-03

Similar Business Software

Plesk

The complete set of features to build, secure and run. Everything you'll need to effortlessly build your projects, secure against vulnerabilities and automate your day-to-day tasks - on one simple dashboard. Add domains and subdomains and move them between subscriptions. Configure DNS and...

See Software
Dekoo

Dekoo is the Asia's independent global search engine company offering Unique search, Unbiased results on the web having its own crawler, indexing, search and ranking algorithms along with AI/ML technologies with Tier 4 Datacenters and Content Delivery Network around the world. It aims to...

See Software
Gigablast

Gigablast is a search engine designed to provide users with efficient and fast search results. It aggregates data from various sources, giving access to web pages, images, and more, allowing for a comprehensive search experience. Gigablast is particularly focused on simplicity and speed,...

See Software
ispmanager

Ispmanager – websites and web environment Linux control panel that makes life easier for web developers and hosting professionals. The main features: - Work and download a popular CMS: WordPress, Drupal, Joomla - Manage files through the built-in file manager - Choose from various web...

See Software
Perplexity

Where knowledge begins. Perplexity is an AI search engine that gives you quick answers. Available for free at as a web app, desktop app, or on the go on iPhone or Android. Perplexity AI is an advanced search and question-answering tool that leverages large language models to provide...

See Software
Exa

The Exa API retrieves the best content on the web using embeddings-based search. Exa understands meaning, giving results search engines can’t. Exa uses a novel link prediction transformer to predict links which match the meaning of a prompt. For queries that need semantic understanding,...

See Software