Heritrix: Internet Archive Web Crawler

We hang in http fetcher if using midFetchAbort filters.

Reported by Tom Emerson. See below.

Tom gave me an order file and seeds to reproduce the
problem with. Happens fairly soon after startup using
his order and seeds.

When hung, thread dump showed we were stuck trying to
get a connection from host connection pool. See below.

Turns out we were not returnign aborted connection to
the connection pool (Didn't think it was necessary
since doing the release after the abort threw ugly
Connection is not Open exceptions).

Added to this issue is a patch that first releases a
connection before calling abort. Does it for the
midfetch, and for timer and length aborts.

Giving to Tom to test.


I'm running a build synched from CVS head this
afternoon. All 50
threads are stuck: here's a subset of the toe threads
report:

Toe threads report - 200411130004
Job being crawled: Vietnamese1
Number of toe threads in pool: 50 (50 active)
ToeThread #1
#1
http://www.saigonnet.vn/homepage-data/tb/2004/tb-taikhoan.htm
(0 attempts)
X http://www.saigonnet.vn/
Current processor: HTTP
ACTIVE for 21m33s576ms
Where: ABOUT_TO_BEGIN_PROCESSOR for 1293575ms
ToeThread #2
#2 http://www.mot.gov.vn/en/index.asp (0 attempts)
L http://www.mot.gov.vn/
Current processor: HTTP
ACTIVE for 21m23s194ms
Where: ABOUT_TO_BEGIN_PROCESSOR for 1283193ms
ToeThread #3
#3
http://203.162.1.217/ASX_01042004/041111canhac_motthoangtaynguyen.wmv
(0 attempts)
ELL
http://vnntelevision.net/VOD/index.asp?offset=10
Current processor: HTTP
ACTIVE for 21m41s48ms
Where: ABOUT_TO_BEGIN_PROCESSOR for 1301047ms

and so on and so on for all 50.

This has happened in two crawls with two different seed
lists.

Is this a problem on my side (we had some network
issues earlier which
were fixed) or is this indicatgive of something else.

-tree