Robots.io is a Java library designed to make parsing a websites 'robots.txt' file easy.

The RobotsParser class provides all the functionality to use robots.io.

Domains passed to RobotsParser are normalised to always end in a forward slash. Disallowed Paths returned will never begin with a forward slash. This is so that URL's can easily be constructed.

Project Activity

See All Activity >

License

GNU General Public License version 3.0 (GPLv3)

Follow Robots.io

Robots.io Web Site

Other Useful Business Software
Custom VMs From 1 to 96 vCPUs With 99.95% Uptime Icon
Custom VMs From 1 to 96 vCPUs With 99.95% Uptime

General-purpose, compute-optimized, or GPU/TPU-accelerated. Built to your exact specs.

Live migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
Try Free
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of Robots.io!

Additional Project Details

Programming Language

Java

Related Categories

Java Search Engines, Java Site Management Software

Registered

2014-12-24