Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-351 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Name->IP mappings in a 'hosts' file aren't technically DNS
lookups, so it's not surprising that DNSJAVA doesn't find them.

For our crawls at the Archive, we only want to consult
public DNS, not any private settings.

However, it's reasonable to be able to fall back on local
settings, esp. if/when crawling an intranet.

So I've added an expert setting to FetchDNS:
'accept-non-dns-resolves'. Default is false -- whcih gets
the same behavior as before, and reported in this bug.
Setting it to true will try InetAddress.getByName() after a
DNS lookup fails, and if the getByName succeeds, use the
resulting IP address, and set the URI's fetch status to the
new code S_GETBYNAME_SUCCESS (1001).

This will allow crawling of the target host to proceed,
though no record of the name->IP binding will be present in
the ARC files.

Commit comment:
Fix for [ 1116204 ] FetchDNS doesn't work (bug in dnsjava)
* FetchStatusCodes.java
Add new code S_GETBYNAME_SUCCESS (1001) - for non-DNS
successful IP lookups
* FetchDNS.java
Add new 'accept-non-dns-resolves' setting. If true,
fallback to InetAddress.getByName()
* ARCWriterProcessor.java
Only try to write text/dns record on true DNS success

Logged In: YES
user_id=924942

Upping priority.

Logged In: YES
user_id=924942

What would you suggest? That when we get back a null, that
we go to InetAddress.getByName?

Maybe I should try this on linux adding an address to
/etc/hosts to see if I can reproduce your windows behavior?