Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-274 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Change committed. Commit comment:

Fix for [ 1055854 ] completed crawls show as 'aborted by user'
* HostQueuesFrontier.java
For test of isEmpty() near end of next(), don't try to
end crawl, just 'continue' through so next
controller.checkFinish() can end things cleanly.

Logged In: YES
user_id=924942

Testing: I see the change take effect because I can see
status for small crawls before patch logged as 'aborted by
user' but post patch, the same crawl has 'finished'. I also
see that if I actually terminate the crawl, I get 'aborted
by user' as I should.

+1 on this patch.

Back to G.

Logged In: YES
user_id=924942

Testing: I see the change take effect because I can see
status for small crawls before patch logged as 'aborted by
user' but post patch, the same crawl has 'finished'. I also
see that if I actually terminate the crawl, I get 'aborted
by user' as I should.

+1 on this patch.

Back to G.

Logged In: YES
user_id=144912

Looking into this, I see that the isEmpty() check and block near
the end of HostQueuesFrontier is causing problems by trying
to end the
crawl itself. In particular, by using
controller.requestCrawlStop(),
a 'finished normally' looks like an 'ended by operator'.

The most simple fix is to let the controller.checkFinish() (at
the top of next()) end things cleanly on the next loop pass.
Thus,
upon detecting an empty frontier, the near-end isEmpty()
check only
needs to make sure there's no extra wiating for state to change,
and it can do this by a 'continue' of the while(true) loop.

(It appears that the only way that the initial checkFinish() can
return false, but the end-of-method isEmpty() return true, is if
a politeness-snoozed queue woke in the call to
wakeReadyQueues(), was empty, and gets discarded. Thus
another potential fix would be to move the wakeReadyQueues()
line
mid-method to before the controller.checkFinish(). Then the
near-end isEmpty() check would be superfluous. However, this
is slightly higher risk, I'll defer such a fix to later.)

Proposed fix, which cleanly 'finishes' my short
HostQueuesFrontiers test crawls that had appeared 'operator
ended' before:

Index: HostQueuesFrontier.java
===================================================================
RCS file:
/cvsroot/archive-crawler/ArchiveOpenCrawler/src/java/org/archive/crawler/frontier/HostQueuesFrontier.java,v
retrieving revision 1.24
diff -u -r1.24 HostQueuesFrontier.java
--- HostQueuesFrontier.java 29 Oct 2004 01:40:52 -0000 1.24
+++ HostQueuesFrontier.java 3 Nov 2004 20:11:07 -0000
@@ -626,11 +626,10 @@
enqueueToKeyed(curi);
}
}
-
+
// See if URIs exhausted
if(isEmpty()) {
- this.controller.requestCrawlStop();
- throw new EndedException("exhausted");
+ continue; // next controller.checkFinish()
will end cleanly
}

if(alreadyIncluded.pending() > 0) {

Logged In: YES
user_id=144912

Changed terminology. 'Finished' crawls showing as 'ended by
operator' only occurs from HostQueuesFrontier... (works
right in BdbFrontier). Investigating.