#7 Adhere to robots.txt file

[Hans de Graaff]

Adhere to the robots.txt standard by using the
LWP::RobotUA instead of the normal UA. I am not sure if
this is the right thing to do, since Checkbot is not an
ordinary robot. The problem is that the robots.txt file
makes no distinction between crawling a site and
indexing it. Checkbot only attempts the first, and then
only when explicitly requested to do so. For this
reason I never saw much use in implementing this.

Note that the Robots META tag does make this
distinction, and support for it has been implemented.

See http://www.robotstxt.org/wc/exclusion.html for more
details.

The attached patch is against Checkbot 1.63 and
implements this support.

[Hans de Graaff]