RobotsDisallowed Files

A curated list of the most common and most interesting robots.txt

This is an exact mirror of the RobotsDisallowed project, hosted at https://github.com/danielmiessler/RobotsDisallowed. SourceForge is not affiliated with RobotsDisallowed.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
RobotsDisallowed source code.tar.gz	2019-07-23	21.1 MB	0
RobotsDisallowed source code.zip	2019-07-23	21.1 MB	1
Totals: 2 Items		42.2 MB	1

Other Useful Business Software

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.

Start Free

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

Recommended Projects

react-cookie-consent
A small, simple and customizable cookie consent bar for use in React
Crawler Detect
CrawlerDetect is a PHP class for detecting bots/crawlers/spiders
MBR Cookie Consent
Enterprise-grade GDPR/CCPA compliant cookie consent with IAB TCF v2.3
RobotsTxt
The repository contains Google's robots.txt parser
Data Harvesting Framework
An extensible framework for the automated harvesting, indexing and collation of publicly available data from internet resources for example RSS feeds and webservices.