Heritrix: Internet Archive Web Crawler

Tracker: Bugs

5 Failed DNS does not have intended impact - ID: 935352

Last Update: Comment added ( karl-ia )

Details:

PreconditionEnforcer.considerDnsPreconditions() is not
having its intended effect after a DNS lookup has tried
and failed: it is not marking the CrawlURI which would
need the DNS lookup as failed.

Instead, when default IP validity is in effect:

- the call in FetchDNS which marks even failed lookups
as having occured supplies a 0 TTL
- when considerDnsPreconditions() later checks, it sees
the IP info as expired rather than tried-and-failed.
Thus, the first 'if' test of considerDnsPreconditions()
schedules a retry before the second 'if' can note that
the DNS has tried-and-failed.

Submitted:

Gordon Mohr ( gojomo ) - 2004-04-15 01:51

Priority:

Status:

Closed

Resolution:

Fixed

Assigned:

Gordon Mohr

Category:

None

Group:

None

Visibility:

Public

Comments ( 2 )

Date: 2007-03-14 00:10 Sender: karl-ia This issue is now discussed in the new JIRA tracker at http://webteam.archive.org/jira/browse/HER-117 -- please add further comments at that location.
Date: 2004-04-15 01:52 Sender: gojomo Logged In: YES user_id=144912 Fix appears to be inverting the order of the two if tests in considerDnsPreconditions: then true DNS failure is noted -- and applied to the dependent CrawlURI -- before scheduling another DNS try when appropriate is considered.

Attached File

Changes ( 3 )

Field	Old Value	Date	By
status_id	Open	2004-04-15 02:00	gojomo
resolution_id	None	2004-04-15 02:00	gojomo
close_date	-	2004-04-15 02:00	gojomo

Heritrix: Internet Archive Web Crawler

Tracker: Bugs

Comments ( 2 )

Attached File

No Files Currently Attached

Changes ( 3 )

About SourceForge

Find Software

Develop Software

Community

Help