Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-243 -- please add further
comments at that location.

Logged In: YES
user_id=924942

Committed a fix though I ain't exactly sure why the commit
fixes observed issue. Commit message below. Will mark it
fixed and open a new issue for any new manifestations of the
behavior seen in this issue -- hopefully the new forms will
give better clue as to what the timing/jsp issue is.

Fix for '[ 1024120 ] Lost crawl job after terminate running
job with jobs
pending'.
* src/java/org/archive/crawler/admin/CrawlJobHandler.java
Formatting. Changed notify to notifyAll so UI gets its
wakeup sooner --
rather than having to wait on timeout. Moved start of
any new pending
jobs to crawlEnded from crawlEnding. This fixes issue
and it looks like it
also fixes problem where when 3 jobs are queue, the last
gets stuck at
startup -- the UI doesn't refresh. I'm not sure exactly
why the move of
start new job to crawlEnded fixes observed issue --
timing, or state in the
jsp -- but I've spent enough time trying to figure it.
We may be bitten by something similar again later.

Logged In: YES
user_id=924942

Here is a related comment from Kris:

Hey Michael,

It occured to me that now that you loose all the reports
etc. when a job finishes/is terminated, we might want to
change the UI so that the console doesn't drop a job until
it is actually ended, rather then in the process of ending.
This would also make it less likely that people terminate a
job and then shut down the application before the job has
had a chance to fully terminate and write it's reports. As
it is you have to monitor the logs to know when it is safe
to shut of Heritrix, if you just terminated a crawl.

- Kris

Logged In: YES
user_id=144912

Seen in HEAD at NLA workshop. 'Second' crawl was active, but
invisible to UI as it proceeded. Eventually, it finished (on
its own), and the third job began, appearing in the UI.
There was no evidence of the 2nd job in the UI after its
completion.

Suspect thread safety/timing issue.