#227 Feature request: Robots exclusion rules parser/checker

[Anonymous]

It would be nice to see an extension that allows
parsing of a webserver's robots.txt file to see if our
client is allowed to access a specific resource. Such
an extension should return a boolean value.

This feature would prove very useful when making web
spiders (in general, web clients requiring no user
interaction).

[Michael Schlenker]

Logged In: YES
user_id=302287

I have partial code that does this. I'll see if i can beat
it into usable shape.