The spider should follow outbound valid links to crawl sites. In the beginning there is a need for some sort of restriction so that we don't try to "download the internet".
Log in to post a comment.
