Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-200 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Not a bug. Recover log is only supposed to approach the
state of the crawler at the time of a problem, not have
resolution of every frontier operation, so losing a little
due to unclosed gzip state at end is OK tradeoff.

Logged In: YES
user_id=144912

recover log should be closed properly on operator
crawl-termination -- but that's not the scenario for which
recover log is designed. Random failure up to whole machine
crash will unavoidably leave gzip stream corrupt at end, but
loss is limited, and recover log isn't supposed to recover
perfectly, for every URI of progress, just save "most" work.

Possibly could flush recover log more often, ensuring lost
end-segment is smaller.

Probably should just doc and close.

Logged In: YES
user_id=924942

So, not the whole log is useless... just the last compressed
32k corrupted block. All before that can be read by the
fronter.