#9 Missing robots.txt

[Anonymous]

I think it would be very useful to include a robots.txt in the dirbuster lists. I see robots-txt and robots_txt, so probabry some translation from the real file has been made which isn't very useful for this project.

[sittinglittleduck]

DirBuster was always intended to find random things on web servers, I have have had a few requests for it to find more default files, So I been working on it using the nikto database to do this. This will be introduced in the next release.

James