When using Storage Implementation other then
InMemoryStorageImpl and when base site
site.robotstxt.fetch set to false only start page is
parsed, all other URL are queued indefinetly.
Reason: in AgentImpl method visit( URL, URLFoundEvent)
when site.getFetchRobotsTXT() returns false Site status
is updated to ROBOTSTXT_SKIPPED but site isn't stored.
It works with InMemory Storage, but any other type of
storage all fetched URLs are queued indefinetly.
Log in to post a comment.