Re: [htdig-dev] robots.txt not working

[Andy L. <al...@ju...>]

Thanks Gilles,

I did over look that user_agent attribute and have added that.

Any chances on speeding up indexing? I had to interupt indexing because of
the slowness.

I can't even index 30k of pages in a 24 hour period. That hurts. Got to
have some speed when indexing.

Regards,

Andy

On Fri, 14 Nov 2003, Gilles Detillieux wrote:

> According to Andy Lewis:
> > Look like the robots.txt file isn't being parsed properly.
> >
> > I've used the
> > <http://www.jumboclassifieds.com/~alewis/attrs.html#robotstxt_name>
> > robotstxt_name tag and added the same name to my robots.txt file and I
> > still see the
> > default htdig name when indexing.
> >
> > Any ideas? Running the lastest beta. Downloaded today.
>
> It seems to me you're confusing the robotstxt_name attribute with
> the user_agent attribute.  If by "I still see the default htdig name"
> you mean that's what's showing up in the access_log, then you want to
> change user_agent.
>
> See http://www.htdig.org/dev/htdig-3.2/attrs.html#user_agent
>
> There is a bug in 3.2.0b5 in that it doesn't correctly handle an empty
> Disallow directive, but that doesn't seem to be the issue here.  The fix
> for this latter bug is at
>
> ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/robots.0
>
> --
> Gilles R. Detillieux              E-mail: <gr...@sc...>
> Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
> Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)
>