It would be nice if Snoopy is able to check the robots.txt and follow it depending on the inserted User-Agent.
Log in to post a comment.
