There is no need for the spider to crawl sites that aren't proper sites. A lot of the links pointing from sites will point to nowhere good. A validation of links for crawl is to be added.
Log in to post a comment.
