Robots.io

Robots.io is a Java library designed to make parsing a websites 'robots.txt' file easy.

The RobotsParser class provides all the functionality to use robots.io.

Domains passed to RobotsParser are normalised to always end in a forward slash. Disallowed Paths returned will never begin with a forward slash. This is so that URL's can easily be constructed.

Project Activity

See All Activity >

License

GNU General Public License version 3.0 (GPLv3)

Follow Robots.io

Robots.io Web Site

Other Useful Business Software

Build Agents and Models on One Platform

Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.

Try It Free

Rate This Project

User Reviews

Be the first to post a review of Robots.io!

Additional Project Details

Programming Language

Java

Related Categories

Java Search Engines, Java Site Management Software

Registered

2014-12-24

Similar Business Software

ispmanager

Ispmanager – websites and web environment Linux control panel that makes life easier for web developers and hosting professionals. The main features: - Work and download a popular CMS: WordPress, Drupal, Joomla - Manage files through the built-in file manager - Choose from various web...

See Software
Google

Our mission is to organize the world's information so that everyone can access and use it. Every time you search, there are thousands, sometimes millions, of webpages with helpful information. How Google figures out which results to show starts long before you even type, and is guided by a...

See Software
Perplexity

Perplexity is an AI-powered search and answer engine designed to provide accurate, real-time information. It combines natural language processing with web search to deliver concise and reliable answers. Users can ask questions conversationally and receive responses backed by cited sources. The...

See Software
aaPanel

aaPanel, a simple but powerful control panel, can manage the web server through web-based GUI(Graphical User Interface). aaPanel provides the one-click function such as one-click install LNMP/LAMP developing environment and software. Our main goal is to help users to save the time of deploying,...

See Software
Swisscows

Since we never collect your data, we never track your data! We respect your privacy! It is in our DNA to protect every user! With Swisscows you are 100% anonymous! Have our own servers and do not work with a cloud or third party! Family is the most important cell of our society! If morality and...

See Software
KeyHelp

You already have the best hardware for your Internet projects, but complicated management, unreliable performance and high costs are causing discontent? With KeyHelp we provide you an amazing server control panel for convenient management of your Linux webserver (for Ubuntu and Debian). You will...

See Software