#8 No activity besides start page when robotstxt.fetch is false

[Dmitri]

When using Storage Implementation other then
InMemoryStorageImpl and when base site
site.robotstxt.fetch set to false only start page is
parsed, all other URL are queued indefinetly.
Reason: in AgentImpl method visit( URL, URLFoundEvent)
when site.getFetchRobotsTXT() returns false Site status
is updated to ROBOTSTXT_SKIPPED but site isn't stored.
It works with InMemory Storage, but any other type of
storage all fetched URLs are queued indefinetly.

[Dmitri]

Fix - line 167 added.