Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-724 -- please add further
comments at that location.

Logged In: YES
user_id=924942

Closing again after making below change.

Refinish of '[ 903093 ] Setting of Integer.MAX_VALUE is ugly'.
Forgot to modify the default order.xml file upon review that
observed that zero for max-referral-hops and embed hops can
legitimately be zero.
* src/conf/profiles/Simple/order.xml
Change the defaults from 0 to -1.

Logged In: YES
user_id=924942

Below was a commit I made subsequent to a Gordon comment
(included below) sugggesting that zero are legit values for
hops. Testing the 120 candidate, looks like I failed to
change the default profile order.xml from 0 to -1. Still
testing. Reopening with high priority for the moment till
testing is over so don't overlook this issue.

revision 1.18
date: 2004/10/12 19:21:55; author: stack-sf; state: Exp;
lines: +9 -9
Change upperbound signifier to -1 from 0.
Change prompted by Gordon review. Here is his comment:

> Sometimes you need to set these values to a true zero,
meaning don't accept
> *any* hops of this type (or of all transclusion types).
So, zero can't be
> used to mean, "no limit".

> (Example: you've fed the crawler a long list of exact URIs
to fetch, where
> no links/embeds/redirects should be followed.)

* src/java/org/archive/crawler/filter/TransclusionFilter.java
Make signifier for no upperbound -1 rather than 0.

Logged In: YES
user_id=924942

Fixed. Here is commit message:

Fix for [ 903093 ] Setting of Integer.MAX_VALUE is ugly
* src/conf/profiles/Simple/order.xml
* src/conf/selftest/order.xml
Set default max-length to be -1.
* src/java/org/archive/crawler/fetcher/FetchHTTP.java
Use default max-length of download of -1.
* src/java/org/archive/io/RecordingInputStream.java
Only test for maxLength limit if value to test is > 0.
* src/java/org/archive/crawler/filter/TransclusionFilter.java
Use 0 as no limit for hops.