Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-446 -- please add further
comments at that location.

Logged In: YES
user_id=1269624

Test in harness has been passing for a week, moved to
regression harness.

Closing bug.

Logged In: YES
user_id=924942

Fix introduced a bug where we'd look for ever on a
(perverse) redirect where the redirect was sending us back
to the page we were redirected from.

E.g.
http://bridalelegance.com/images/buttons3/tuxedos-off.gif
has a 'Content-Location' of itself. We'd redirect, then
redirect, then redirect... to this URL over and over again.

Be careful w/ that forcefetch.


Fix bug introduced by fix to '[ 1222360 ] strip-www canon.
rule causes failed
crawl of netarkivet.dk'
* src/java/org/archive/crawler/frontier/AbstractFrontier.java
If redirect and we're redirecting to ourselves, don't
forcefetch else
we'll loop endlessly.

Logged In: YES
user_id=924942

Gordon:

Thanks for adding reference to history (I should have reused
the original issue).

Yes, latest changes equate to parenthetical 'or force-fetch'
from that issue .

Yes, there may still be gaps. Does 'Chakrabarti/Mining the
Web' have a canonicalization section? I don't remember (Do
you have this book? I haven't seen it in a while). Was
thinking a review of alexa rules might help too.

Logged In: YES
user_id=144912

It would be better if this issue was closed as a duplicate
of "[ 1078094 ] www-strip canonicalization unintended
exclusion of redirect"-- and these reports/comments/fixes
documented there:
http://sourceforge.net/tracker/index.php?func=detail&aid=1078094&group_id=73833&atid=539099

The 'Summary' there perfectly fits this problem; this issue
is best understood in the context of previous similar work;
and I *think* the effect of the latest changes is
essentially equivalent to the parenthetical of my first fix
suggestion in that issue: if the 'via' and 'uri'
canonicalize to the same thing, set force-fetch.

I had reservations about the suggestion then as not covering
all cases... it may still have gaps now.

In particular, we still have a canonicalization problem when
A and B canonicalize to the same thing, but A succeeds while
B fails. If B is encountered first, variant A will never be
crawled.

This could also happen in a (as yet contrived but plausible)
2-hop redirect:

- crawler encounters http://host.com/
- redirect is to http://host.com/index.html
- redirect is to http://www.host.com

We should review the literature (Chakrabarti/Mining the Web)
and other crawlers to see if there's a more comprehensive
fix to our canonicalization issues.

Logged In: YES
user_id=924942

Reopening and assigning to Karl for verification of fix.

Logged In: YES
user_id=924942

Committed fix. Below is commit message.

Fix for '[ 1222360 ] strip-www canon. rule causes failed
crawl of
netarkivet.dk'.
* src/java/org/archive/crawler/frontier/AbstractFrontier.java
Replaced conditionalCanonicalization with method
canonialization that
takes a CandidateURI. New method sets forceFetch if
canonicalization
of a via == current URL AND this URL is result of
redirect. Previous, for
this special circumstance, we'd return the
non-canonicalized version of
the URL. This non-canonicalized URL would pass through
the already seen
check and the URL would be crawled. This worked for case
where
non-canonicalized form did not equal the via -- e.g. if
site was trying to
redirect us from DOMAIN to www.DOMAIN (via ==
canonicalized form). The
technique wouldn't work where we were being redirect
from www.DOMAIN to
DOMAIN (via != canonicalized form && non-canonicalized
form == via).
* src/java/org/archive/crawler/frontier/WorkQueueFrontier.java
Refactor schedule. Call canonicalize. Then switch off
forceFetch flag
(Canonicalize has changed. It may set forcefetch).