Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-881 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Situation was improved by previous fixes; other needs should
be raised as new issues if/when need/plan is clearer.

Logged In: YES
user_id=924942

Applied suggested patch with minor mod (Changed
'sendCrawlStateChangeEvent(state, state.toString());' to
'sendCrawlStateChangeEvent(state, jobState);').

Below is commit.

Lowered priority.

Moved issue to RFE database. Lets address crawl job and
crawl controller state confusion refactoring as an RFE.

Palliative for '[ 1104696 ] Confusion: CrawlController and
CrawlJob States'
Below is a Gordon patch.
* src/java/org/archive/crawler/framework/CrawlController.java
(sendCrawlStateChangeEvent): On startup, pass in
CrawlJob.STATUS_* rather than CrawlController.* status
as message on startup. Previous we passed in
CrawlController state which confused CrawlJob at start time.

Logged In: YES
user_id=1220421

I have just suffered from this bug. Are there any objections
against merging the suggested patch into CVS HEAD?

Christian

Logged In: YES
user_id=144912

Long-term, there should be a clear distinction between
CrawlController (API to an abstract 'crawl') and CrawlJob
(UI/workflow object). CrawlController would have no
references to CrawlJob, with CrawlJob just one possible
client of CrawlController's generic capabilities.

Currently, it looks like the intent of the code is for the
'newState' argument of sendCrawlStateChangeEvent() to be
drawn from CrawlController states, and the 'message'
argument drawn from CrawlJob statuses. The code respects
this split, *except* in requestCrawlStart(), where the
CrawlController states are used in both positions.

So a potential fix that delays larger refactoring would be
to harmonize requestCrawlStart()'s use of
sendCrawlStateChangeEvent() with the uses. Here's a patch
that should do the trick:

Index: CrawlController.java
===================================================================
RCS file:
/cvsroot/archive-crawler/ArchiveOpenCrawler/src/java/org/archive/crawler/framework/CrawlController.java,v
retrieving revision 1.90
diff -u -r1.90 CrawlController.java
--- CrawlController.java 11 Jan 2005 03:55:51 -0000 1.90
+++ CrawlController.java 21 Jan 2005 09:19:41 -0000
@@ -835,8 +835,16 @@

// Assume Frontier state already loaded.
logger.info("Starting crawl.");
- sendCrawlStateChangeEvent(STARTED, STARTED.toString());
- state = beginPaused ? PAUSED : RUNNING;
+
+ sendCrawlStateChangeEvent(STARTED,
CrawlJob.STATUS_PENDING);
+ String jobState;
+ if(beginPaused) {
+ state = PAUSED;
+ jobState = CrawlJob.STATUS_PAUSED;
+ } else {
+ state = RUNNING;
+ jobState = CrawlJob.STATUS_RUNNING;
+ }
sendCrawlStateChangeEvent(state, state.toString());
// A proper exit will change this value.
this.sExit = CrawlJob.STATUS_FINISHED_ABNORMAL;