Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-858 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Pause-at-start option implemented. Commit comment:

Implementation of [ 1069331 ] hold paused crawl at 'end',
allowing all in-progress ops (pause-at-start addition)
* Frontier.java
add 'start()' method as specific place to do crawl-start
activity
* CrawlController.java
have crawl start use frontier.start(); remove vestigial
previous attempt at begin-paused functionality; have
requestCrawlPause() complete-the-pause if no threads are
in-progress
* AbstractFrontier.java
add 'pause-at-start' option (default false); when set,
start() does a requestCrawlPause() rather than unpause() to
begin
* AdaptiveRevisitFrontier.java, HostQueuesFrontier.java
add default start() implementation: just unpause
* CrawlJobHandler.java
remove redundant explicit setting of job status that
introduced a bug when CrawlController.requestCrawlPause()
can complete the pause immediately

Assigning to Karl for verification.

Logged In: YES
user_id=144912

Fixed for crawl end; commit comment:

Implementation of [ 1069331 ] hold paused crawl at 'end',
allowing all in-progress ops
* CrawlController.java
separate atFinish() test from checkFinish() operation;
raise visibility of beginCrawlStop; improve comments
* AbstractFrontier.java
add pause-at-finish setting; if set, only truly 'end'
crawl if finish condition is met coming out of a pause.
Otherwise, treat finish condition as request to pause.

Leaving open at lower priority to add pause-at-crawl-start
capability.

Logged In: NO

I've been thinking we need something like this at crawl
start too -- that you'd set up an order, start it, and
though there is nothing currently to crawl, that the crawler
would sit in a 'ready-to-crawl' holding pattern waiting on
submission of URLs to crawl (While in this configuration, it
wouldn't quit until it got a shutdown message). We'd use
'ready-to-crawl' state in a multimachine configuration: We'd
ready the hive of crawlers by putting them all into the
'ready-to-crawl' configuration and we'd then start injecting
URLs into the cluster.