#103 improve fail-over with inexisting DNS hosts/add audit info

[Ralf Hauser]

Asynchronous questioning of my entire resolver array is an excellent strategy if my primary DNS is unreachable or fails to provide an answer.

Are there any examples what such a ResolverListener should look like?

I am happy to test!!

[Brian Wellington]

Logged In: YES
user_id=63197
Originator: NO

The ExtendedResolver class already supports querying multiple DNS servers; I'm not sure what you're looking for that isn't there.

[Ralf Hauser]

Logged In: YES
user_id=266141
Originator: YES

Hi Brian,

Somehow, querying multiple servers doesn't work if I have two DNS servers and the first one is wrong.

String[] serversArrayTest = { "194.230.1.168", "194.246.118.118" };

If I change the order, with the correct dns server first, it does work! So, perhaps I need to set another parameter to have the ExtendedResolver do a retry with a different server upon failure?

If I set resolver.setLoadBalance(true);
and execute 5 arbitrary domain MX lookups, every other lookup works.

Any idea what is wrong?

That's why I came up with the idea to at least try to asynchronously fill the cache with the other servers if such a failure occurs...

Ralf

My test-case is derived from how apache james uses dnsjava-2.0.3.

            static ExtendedResolver resolver = new ExtendedResolver\(serversArray\);
            resolver.setTimeout\(resolverTimeOutInSecs\);
            // resolver.setLoadBalance\(true\);
            log.debug\("set resolver: " + resolver\);
        \}
        Record answers\[\] = rawDNSLookup\(hostname,false, Type.MX\);
        if \(answers == null\) \{
            log.info\("answers == null"\);
            return servers;
        \}
        MXRecord mxAnswers\[\] = new MXRecord\[answers.length\];
        for \(int i = 0; i < answers.length; i++\) \{
            mxAnswers\[i\] = \(MXRecord\) answers\[i\];
        \}



private static Record\[\] rawDNSLookup\(String namestr, boolean querysent, int type\) throws DefaultException \{
    Name name = null;
    try \{
        name = Name.fromString\(namestr, Name.root\);
    \} catch \(TextParseException tpe\) \{
        log.error\("Couldn't parse name " + namestr, tpe\);
        return null;
    \}
    int dclass = DClass.IN;

    SetResponse cached = cache.lookupRecords\(name, type, dnsCredibility\);
    Record question = null;
    if \(cached.isSuccessful\(\)\) \{
        log.debug\(new StringBuffer\(256\).append\("Retrieving MX record for "\).append\(name\).append\(" from cache"\)
                .toString\(\)\);

        return processSetResponse\(cached\);
    \} else if \(cached.isNXDOMAIN\(\) || cached.isNXRRSET\(\)\) \{
        return null;
    \} else if \(querysent\) \{
        return null;
    \} else \{
        log.debug\(new StringBuffer\(256\).append\("Looking up MX record for \""\).append\(name\).toString\(\) + "\", type "
                \+ type + ", dclass " + dclass\);
        question = Record.newRecord\(name, type, dclass\);
        log.debug\("question = \"" + question.toString\(\) + "\""\);
        Message query = Message.newQuery\(question\);
        // log.debug\("query = \"" + query.toString\(\) + "\""\);
        Message response = null;
        long elapsed = 0;
        try \{
            response = resolver.send\(query\);
        \} catch \(Exception ex\) \{
            log.warn\("Query error for question " + question.toString\(\) + ", name " + name + ", type " + type
                    \+ ", dclass " + dclass + ", timeout in sec " + resolverTimeOutInSecs + ", ex = "
                    \+ ex.getMessage\(\) + ", elapsed = " + elapsed, ex\);
            if \(resolver == null\) \{
                log.error\("resolver == null", new Exception\(\)\);
                throw new DefaultException\("resolver == null" + ", ...
            \}
            return null;
        \}
        if \(response == null\) \{
            if \(log.isDebugEnabled\(\)\) \{
                log.debug\("response==null - possibly due to timeout" + ", timeout in sec " + resolverTimeOutInSecs
                        \+ ", elapsed = " + elapsed\);
            \}
            return null;
        \}
        log.debug\("response found"\);
        int rcode = response.getHeader\(\).getRcode\(\);
        if \(rcode == Rcode.NOERROR || rcode == Rcode.NXDOMAIN\) \{
            cached = cache.addMessage\(response\);
            if \(cached \!= null && cached.isSuccessful\(\)\) \{
                return processSetResponse\(cached\);
            \}
        \}
        if \(rcode \!= Rcode.NOERROR\) \{
            log.debug\("rcode = " + rcode\);
            return null;
        \}
        return rawDNSLookup\(namestr, true, type\);
    \}
\}

[Brian Wellington]

Logged In: YES
user_id=63197
Originator: NO

I don't know what you mean by "the first one is wrong". The ExtendedResolver follows the standard resolution protocol - if there's a timeout from the first server, or it gives an answer that's clearly broken, the second server is contacted. If the first server gives an answer which is a valid DNS response, the second server isn't contacted.

[Ralf Hauser]

Logged In: YES
user_id=266141
Originator: YES

Hi Brian,

It appears that the first server in my testcase creates a timeout, but the second server is not contacted?
Is it possible that the rule you describe doesn't apply for MX records?

Pls have a try with my little test case.

Ralf

5 hauser@pc:~/workspace $ ping 194.230.1.168

Pinging 194.230.1.168 with 32 bytes of data:

Request timed out.

Ping statistics for 194.230.1.168:
Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

[Brian Wellington]

Logged In: YES
user_id=63197
Originator: NO

Can you please provide a test case that compiles? There appear to be a lot of references to classes defined in james or elsewhere not in dnsjava or the java standard library, problems which may have been caused by copy-and-paste errors, and other things which just look wrong.

[Ralf Hauser]

Logged In: YES
user_id=266141
Originator: YES

Brian, the problem is fixed on our end. The issue was that we killed the dnsjava thread after several seconds, but (shorter than the "resolverTimeOutInSecs"). But still, two improvements on the dnsjava side would have helped me:
1) even if the dns server is not existing by numeric ip (~UnknownHostException), dnsjava appears to wait until the end of the timeout until it starts with another dns server. This doesn't make sense to me since contrarily to a connection to an existing host not getting through, with an inexisting server, there is no reason to wait at all before failing over?
2) For audit trail completeness, it would be great to extend org.xbill.DNS.Message with an extra field that carries with it which of the possible dns servers
from the constructor's array really provided the info...

What do you think?

Ralf

[Ralf Hauser]

Logged In: YES
user_id=266141
Originator: YES

ooops, perhaps I am mistaken on 1) since it may not be possible to distinguish inexistant ip and unreachable ip on the numeric ip level... :(

[Ralf Hauser]

Logged In: YES
user_id=266141
Originator: YES

hmm, so if I have n dns servers in the array, what is the recommended timeout strategy to ensure that all of those servers are at least tried once before declaring defeat? Do I just have to wait N*timeout?
Or does this mandate ResolverListener all the same where that can be parallelized - e.g. as soon as the first one fails?

[Ralf Hauser]

Logged In: YES
user_id=266141
Originator: YES

as per suggestion 2) if I knew which of my dns servers did answer, I might reshuffle the servers array for the next query to improve my odds for a quick answer

[Ralf Hauser]

Logged In: YES
user_id=266141
Originator: YES

further refinement of item 1)
a) if the host exists, but no dns server runs on it, dnsjava waits until the timeout until the next server is taken.
b) if there is an java.net.UnknownHostException: ns3.interway.ch: ns3.interway.ch
at java.net.InetAddress.getAllByName0(Unknown Source)
at java.net.InetAddress.getAllByName0(Unknown Source)
at java.net.InetAddress.getAllByName(Unknown Source)
at java.net.InetAddress.getByName(Unknown Source)
at org.xbill.DNS.SimpleResolver.<init>(SimpleResolver.java:56)
at org.xbill.DNS.ExtendedResolver.<init>(ExtendedResolver.java:281)
for the server in the first position, other valid server will not be ever consulted.

I added some robustness for this:

public static ExtendedResolver makeResolver(String[] dnsSrvrsArray) {
ExtendedResolver r = null;
boolean success = false;
while (!success && null != dnsSrvrsArray && 0 != dnsSrvrsArray.length) {
try {
r = new ExtendedResolver(dnsSrvrsArray);
success = true;
} catch (UnknownHostException e) {
if (1 == dnsSrvrsArray.length) {
log.error(e.getMessage(), e);
return null;
}
String[] serversArrayFixed = new String[(dnsSrvrsArray.length - 1)];
int j = 0;
for (int i = 0; i < dnsSrvrsArray.length; i++) {
String server = dnsSrvrsArray[i];
String eMsg = e.getMessage();
if (-1 != eMsg.indexOf(server)) {
log.info("dropping inexistant dns server \"" + server + "\"" , e);
} else {
serversArrayFixed[j] = server;
j++;
}
}
dnsSrvrsArray = serversArrayFixed;
}
}
return r;
}

what do you think?

[Ingo]

Ticket moved from /p/dnsjava/feature-requests/9/