lonecrow - 2006-03-18

Hi, I use dotLucene and your spider works very well with it.  However, there is no robots.txt parser in it. We're you planning to add one?

I am going to have a go at adding robots.txt support to it but I am a bit new to multi-threading so I hope I don't mess it up :)