Robots.io is a Java library designed to make parsing a websites 'robots.txt' file easy.
The RobotsParser class provides all the functionality to use robots.io.
Domains passed to RobotsParser are normalised to always end in a forward slash. Disallowed Paths returned will never begin with a forward slash. This is so that URL's can easily be constructed.