Hi, I use dotLucene and your spider works very well with it. However, there is no robots.txt parser in it. We're you planning to add one?
I am going to have a go at adding robots.txt support to it but I am a bit new to multi-threading so I hope I don't mess it up :)
Log in to post a comment.