Robots.io is a Java library designed to make parsing a websites 'robots.txt' file easy.
The RobotsParser class provides all the functionality to use robots.io.
Domains passed to RobotsParser are normalised to always end in a forward slash. Disallowed Paths returned will never begin with a forward slash. This is so that URL's can easily be constructed.
License
GNU General Public License version 3.0 (GPLv3)
You Might Also Like
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of Robots.io!