J-Obey is a Java Library/package, which allows people writing their own crawlers to have a stable Robots.txt parser, if you are writing a web crawler of some sort you can use J-Obey to take out the hassle of writing a Robots.txt parser/intrepreter.

Project Samples

Project Activity

See All Activity >

License

GNU General Public License version 2.0 (GPLv2)

Follow J-Obey (Robots.txt Crawler Module)

J-Obey (Robots.txt Crawler Module) Web Site

Other Useful Business Software
Host LLMs in Production With On-Demand GPUs Icon
Host LLMs in Production With On-Demand GPUs

NVIDIA L4 GPUs. 5-second cold starts. Scale to zero when idle.

Deploy your model, get an endpoint, pay only for compute time. No GPU provisioning or infrastructure management required.
Start Free
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of J-Obey (Robots.txt Crawler Module)!

Additional Project Details

Intended Audience

Developers

Programming Language

Java

Related Categories

Java Search Engines, Java Web Scrapers

Registered

2006-03-30