Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-387 -- please add further
comments at that location.

Logged In: YES
user_id=924942

Did some more work on this issue:

Another detail on '[ 1177462 ] RIS#readFullyOrUntil
IOE/timeout' prompted
by mail on this list this morning asking about Heritrix and
proxy's.
* src/java/org/archive/httpclient/HttpRecorderMethod.java
(handleAddProxyConnectionHeader): Add new method to
handle the
addProxyConnectionHeader. Changes 'keep-alive' directive
to 'close' until
we have support for keep-alive..
* src/java/org/archive/httpclient/HttpRecorderGetMethod.java
* src/java/org/archive/httpclient/HttpRecorderPostMethod.java
Overrided addProxyConnectionHeader. Pass handling to
HttpRecorderMethod#handleAddProxyConnectionHeader.

Logged In: YES
user_id=924942

Bad URL is actually
http://www.odpm.gov.uk/stellent/groups/odpm_control/documents/homepage/odpm_home_page.hcsp

Here is a fix. Here is commit.

Fix for '[ 1177462 ] RIS#readFullyOrUntil IOE/timeout'
* src/java/org/archive/crawler/fetcher/FetchHTTP.java
Added more help text to timeout attributes.
* src/java/org/archive/io/RecordingInputStream.java
Implement FIXME. When server is doing keep-alive, even
though we asked
it not too, we were hanging on the socket timeout and
then when it
expired, throwing an IOException killing the fetch. Do
as FIXME suggests
and on socket timeout, rather than IOE, fall into the
overall timeout
check.