I collecting log files from many servers and merged with clfmerge, for a pan-domain application: the robots access record might start with domain name
"GET http://foo.bar.com/robots.txt 200 14"
so I tried to remove the rule: must start with /
< my $regrobot = qr/^\/robots.txt$/i;
my $regrobot = qr/\/robots\.txt$/i;