Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-315 -- please add further
comments at that location.

Logged In: YES
user_id=144912

That would be intuitive behavior. It runs into two problems:

(1) DNS URIs aren't giving out any 'host' to be used for
classic scoping. UURI.getHost() didn't return anything for
DNS URIs by design; UURI.getReferencedHost() is necessary to
get either the 'contacted' host (as in HTTP) or 'referenced'
host (as in DNS URIs). Classic scoping should be using
getReferencedHost() rather than getHost().

(2) max-trans-hops is being enforced as a cap on consecutive
hops-ending non-link hops, even when the URI is otherwise in
scope. That's probably wrong; max-trans-hops is meant to be
the max hops at which a URI can get a transitive ACCEPT, not
a threshold beyond which a URI gets a hard REJECT. (Much
easier to express in decide-rule terms.)

I've made these changes to effect the intuitive behavior.
There could be a side-effect in that in-focus junk that's
reached via long non-link (trans) hop chains that used to be
ruled out will now be ruled in, but that's better than
ruling out otherwise in-focus material simply because its
hop path ended with too many nonlink hops.

Commit comment:

Fix for [ 1090564 ] max-trans-hops=0 generates -63 in crawl.log
* CrawlScope.java
use getReferencedHost() for isSameHost() test
* ClassicScope.java
don't enforce max-trans-hops as part of exceedsMaxHops
* UURI.java
use getReferencedHost() in cacheHostBasename()

After change, a crawl with max-trans-hops has the expected
behavior.

Closing as fixed.