Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-532 -- please add further
comments at that location.

Logged In: YES
user_id=1269624

Closing; separate RFE for better disk-full Heritrix behaviour.

Logged In: YES
user_id=144912

ARCWriterProcessor already catches and logs (as a
localizedError) IOExceptions during writing. Made a 1-line
change which ensures all localizedErrors also cause an
annotation to be added to the current CrawlURI. Format is:

le:[short-name-of-throwable]@[current-processor-name]

This does not specifically mark crawl.log line with 'disk
full' note -- the annotation may look the same for other
IOExceptions -- but I suspect even that generic markup will
be sufficient for the operator need.

Commit comment:
Fix for [ 1395637 ] crawl.log entires do not reflect 'no
space left' error
* CrawlURI.java
add log annotation for all localizedErrors, of form
"le:[throwable]@[processor]"

We should definitely have a 'fail-gracefully-when-disk-full'
test case (or several). Possible approach: create a temp ram
disk, have crawling write ARCs there, ensure quick failure
(after at least a few successful writes) is reported properly.

Assigning to Karl for implementation of such a test, time
permitting.

Logged In: YES
user_id=144912

Annotation seems better: the code is primarily meant as a
'fetchStatusCode' (though in the case of major unhandled
errors, we've overloaded the code with meanings drawn from
non-fetch conditions).