Re: [htdig-dev] robots.txt not working

[Gilles D. <gr...@sc...>]

According to Andy Lewis:
> Look like the robots.txt file isn't being parsed properly.
>  
> I've used the
> <http://www.jumboclassifieds.com/~alewis/attrs.html#robotstxt_name>
> robotstxt_name tag and added the same name to my robots.txt file and I
> still see the
> default htdig name when indexing.
>  
> Any ideas? Running the lastest beta. Downloaded today.

It seems to me you're confusing the robotstxt_name attribute with
the user_agent attribute.  If by "I still see the default htdig name"
you mean that's what's showing up in the access_log, then you want to
change user_agent.

See http://www.htdig.org/dev/htdig-3.2/attrs.html#user_agent

There is a bug in 3.2.0b5 in that it doesn't correctly handle an empty
Disallow directive, but that doesn't seem to be the issue here.  The fix
for this latter bug is at

ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/robots.0

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)