Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-255 -- please add further
comments at that location.

Logged In: YES
user_id=144912

This fixed, plus some related subtle bugs around the retry
process in BdbFrontier (like fetchAttempts increments being
lost, so max-retries could never be met). Commit comment:

Fix for [ 1043251 ] better/longer dns retries on lookup failure
* AbstractFrontier.java
Add S_DOMAIN_UNRESOLVABLE to statuses requiring standard
retry delay
* BdbFrontier.java
When an item is retried, flush its state to underlying
queue, but only after clearing transient processing state.
* BdbWorkQueue.java
Refactor enqueue() to enable
flush-without-count-increment. (Important to keep running
fetchAttempts counter up-to-date.) Also, synchronize
count-changing operations, as accurate count is crucial to
operation, and increment/decrement is not atomic.
* PreconditionEnforcer.java
Ensure DNS URIs are never considered to need a DNS
prereq (the cause of -6 dns failures observed in FR crawl)

Logged In: YES
user_id=144912

Reopening:

Igor reports on FR test crawls, DNS errors are coming
through as -6 rather than -1, not being retried at all, and
thus the seed is also failing as -6 with no retries.

Logged In: YES
user_id=144912

In HEAD, went further and split S_PREREQUISITE_FAILURE code
into 4 more specific codes:

/** DNS prerequisite failed, precluding attempt */
public static final int S_DOMAIN_PREREQUISITE_FAILURE = -6;

/** Robots prerequisite failed, precluding attempt */
public static final int S_ROBOTS_PREREQUISITE_FAILURE =
-61;
/** DNS prerequisite failed, precluding attempt */
public static final int S_OTHER_PREREQUISITE_FAILURE = -62;
/** DNS prerequisite failed, precluding attempt */
public static final int
S_PREREQUISITE_UNSCHEDULABLE_FAILURE = -63;

If/when the seeds report can be upgraded to better report on
items in progress, or when fetching a URI ultimately fails,
these will help give a better indication of exactly what
went wrong.

Commit comment:

Fix & more for [ 1043251 ] better/longer dns retries on
lookup failure
* AbstractFrontier.java, HostQueuesFrontier.java
Makes S_DOMAIN_UNRESOLVABLE a retryable failure
* FetchStatusCodes.java, CrawlURI.java
Split S_PREREQUISITE_FAILURE into 4 distinct codes, for
domain/robots/other/unscheduleable failures
* PreconditionEnforcer.java, FetchHTTP.java, Postselector.java
Use new, more-specific codes
* user_manual.xml
Document updated codes

Logged In: YES
user_id=144912

Fix applied. Commit comment:

Fix for [ 1043251 ] better/longer dns retries on lookup failure
* Frontier.java
Add S_DOMAIN_UNRESOLVABLE to those statuses triggering a
delayed retry.

Logged In: YES
user_id=144912

Proposed fix works, but makes seeds-report somewhat less
useful: seeds whose DNS prereq are still being tried are
indistinguishable from those which DNS-resolved, but no HTTP
server answered for the URL or its robots.

Treating this as an acceptable limitation for now.

Logged In: YES
user_id=144912

Believe least-risk fix is to add S_DOMAIN_UNRESOLVABLE to
the retryable codes in Frontier.needRetrying(), so that DNS
failures get as many retries, with as much delay between
them, as HTTP connection failures.

Logged In: YES
user_id=144912

Changing to a top-priority bug, because current crawler
behavior is: once DNS fails, that host is marked as both
looked-up, and failed -- and from that point on, all URIs on
that host will fail. This happens even if DNS succeeded
earlier. So you can have a situation where, when DNS info
expires, and a relookup is attempted, a lookup failure
spoils all the still-pending deep URIs on the site.

This bit some long-running crawls today during a temporary
net outage.

Logged In: YES
user_id=924942

Knocking down to normal priority. We have a workaround for
the immediate need; looking through logs for failed lookups
trying those that failed in a later crawl.