Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.
Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
Start Free
Veeam Data Platform v13.1 - Get Your Free Trial
Secure by design, portable by default. Recover clean, fast, anywhere. Start a free trial.
Try Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
Robots.io is a Java library designed to make parsing a websites 'robots.txt' file easy.
The RobotsParser class provides all the functionality to use robots.io.
Domains passed to RobotsParser are normalised to always end in a forward slash. Disallowed Paths returned will never begin with a forward slash. This is so that URL's can easily be constructed.