|
From: Andy L. <al...@ju...> - 2003-11-14 19:00:20
|
Thanks Gilles, I did over look that user_agent attribute and have added that. Any chances on speeding up indexing? I had to interupt indexing because of the slowness. I can't even index 30k of pages in a 24 hour period. That hurts. Got to have some speed when indexing. Regards, Andy On Fri, 14 Nov 2003, Gilles Detillieux wrote: > According to Andy Lewis: > > Look like the robots.txt file isn't being parsed properly. > > > > I've used the > > <http://www.jumboclassifieds.com/~alewis/attrs.html#robotstxt_name> > > robotstxt_name tag and added the same name to my robots.txt file and I > > still see the > > default htdig name when indexing. > > > > Any ideas? Running the lastest beta. Downloaded today. > > It seems to me you're confusing the robotstxt_name attribute with > the user_agent attribute. If by "I still see the default htdig name" > you mean that's what's showing up in the access_log, then you want to > change user_agent. > > See http://www.htdig.org/dev/htdig-3.2/attrs.html#user_agent > > There is a bug in 3.2.0b5 in that it doesn't correctly handle an empty > Disallow directive, but that doesn't seem to be the issue here. The fix > for this latter bug is at > > ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/robots.0 > > -- > Gilles R. Detillieux E-mail: <gr...@sc...> > Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/ > Dept. Physiology, U. of Manitoba Winnipeg, MB R3E 3J7 (Canada) > |