#8 No activity besides start page when robotstxt.fetch is false

open
None
5
2004-07-22
2004-07-19
Dmitri
No

When using Storage Implementation other then
InMemoryStorageImpl and when base site
site.robotstxt.fetch set to false only start page is
parsed, all other URL are queued indefinetly.
Reason: in AgentImpl method visit( URL, URLFoundEvent)
when site.getFetchRobotsTXT() returns false Site status
is updated to ROBOTSTXT_SKIPPED but site isn't stored.
It works with InMemory Storage, but any other type of
storage all fetched URLs are queued indefinetly.

Discussion

  • Dmitri

    Dmitri - 2004-07-19

    Fix - line 167 added.

     
  • Günther Van Roey

    • assigned_to: nobody --> vanrogu
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks