Robots.io is a Java library designed to make parsing a websites 'robots.txt' file easy.

The RobotsParser class provides all the functionality to use robots.io.

Domains passed to RobotsParser are normalised to always end in a forward slash. Disallowed Paths returned will never begin with a forward slash. This is so that URL's can easily be constructed.

Project Activity

See All Activity >

License

GNU General Public License version 3.0 (GPLv3)

Follow Robots.io

Robots.io Web Site

Other Useful Business Software
Get Avast Free Antivirus with 24/7 AI-powered online scam detection Icon
Get Avast Free Antivirus with 24/7 AI-powered online scam detection

Get protection for today’s online threats. Free.

Award-winning antivirus protection, as well as protection against online scams, dangerous Wi-Fi connections, hacked accounts, and ransomware. It includes Avast Assistant, your built-in AI partner, which gives you help with suspicious online messages, offers, and more.
Free Download
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of Robots.io!

Additional Project Details

Programming Language

Java

Related Categories

Java Search Engines, Java Site Management Software

Registered

2014-12-24