Robots.io is a Java library designed to make parsing a websites 'robots.txt' file easy.
The RobotsParser class provides all the functionality to use robots.io.
Domains passed to RobotsParser are normalised to always end in a forward slash. Disallowed Paths returned will never begin with a forward slash. This is so that URL's can easily be constructed.
License
GNU General Public License version 3.0 (GPLv3)Other Useful Business Software
Get Avast Free Antivirus with 24/7 AI-powered online scam detection
Award-winning antivirus protection, as well as protection against online scams, dangerous Wi-Fi connections, hacked accounts, and ransomware. It includes Avast Assistant, your built-in AI partner, which gives you help with suspicious online messages, offers, and more.
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of Robots.io!