Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-1022 -- please add further
comments at that location.

Logged In: YES
user_id=924942

Closing. Works for Eric. See below.

Date: Tue, 22 Aug 2006 16:23:37 -0500
From: Eric <ej@ir.iit.edu>
To: Michael Stack <stack@archive.org>
Subject: Re: Does this work for you?
Message-ID: <20060822212337.GA10653@duvel.ir.iit.edu>

...I tested it by creating a new job based on the default
profile (which had the appropriate classes, so i guess it is
a windows problem) and my depth-first code worked fine.

Logged In: YES
user_id=144912

ping eric

Logged In: YES
user_id=924942

Patch applied with below commit message.

Eric, would you mind testing that it works for you? If so,
make a note in here and I'll close the issue as implemented.

Thanks.

[ 1523276 ] (contrib) Support depth-first search priority
scheduling
Contributed by Eric C. Jensen ecjensen at users.sourceforge.net
Reviewed and amended -- removed tabs, and had to refactor
since patch
was against version 1.6 rather than HEAD 1.7. Did not test
(Default
settings does not change crawler behavior).
* src/java/org/archive/crawler/frontier/AbstractFrontier.java
From Eric:

That part of the patch makes sure to only set seed
redirect
priorities to MEDIUM if they were previously NORMAL
(previously this was done unconditionally). This is
necessary since when the depth-first option is set, seed
redirects will be given HIGH priority before this so we
don't want to lower it to MEDIUM. As conjectured in my
initial comment, some may also want a seperate option to
make redirects (or just seed redirects) relatively one
higher priority than their parent (so they're crawled
immediately rather than waiting for the rest of the
seeds).
* src/java/org/archive/crawler/postprocessor/LinksScoper.java
(ATTR_PREFERENCE_DEPTH_HOPS): Added.
(getSchedulingFor): Amended API passing CrawlURI (Used
logging).

Logged In: YES
user_id=705615

That part of the patch makes sure to only set seed redirect
priorities to MEDIUM if they were previously NORMAL
(previously this was done unconditionally). This is
necessary since when the depth-first option is set, seed
redirects will be given HIGH priority before this so we
don't want to lower it to MEDIUM. As conjectured in my
initial comment, some may also want a seperate option to
make redirects (or just seed redirects) relatively one
higher priority than their parent (so they're crawled
immediately rather than waiting for the rest of the seeds).

Logged In: YES
user_id=924942

Good stuff Eric.

Patch looks good except, why did you do this:

+ if (curi.getSchedulingDirective() ==
CandidateURI.NORMAL)
+
curi.setSchedulingDirective(CandidateURI.MEDIUM);


I don't see how it improves over what was there previously.

Logged In: YES
user_id=705615

As conjectured, promoting pre-req's to priorities 2 less
than their parent is unnecessary, 1 is fine. Fixed patch
attached.

Also, some might want redirects to be given priority 1 less
than their parent to ensure they're crawled before remaining
seeds (another aspect of depth-first), but this should
probably be a seperate option.