#1787 Exessive number of DNS lookups caused by java.net.URL#sameFile

[Sebastian Cato]

Hello,

WebClient#getPage, WebClient#download, html.BaseFrameElement#isAlreadyLoadedByAncestor uses java.net.URL#sameFile for URL comparisons.

java.net.URL#sameFile does two DNS lookups (A, AAAA record) on many systems when comparing two equal URLs, or URLs that are equal by scheme, port and path. It then does comparison based on the resolved addresses.

This causes a lot of DNS lookups to be performed (sometimes up around ~100K per run for my problem). Since this is a blocking I/O operation, DNS latency slows things down, especially when there's no other network data being transferred (e.g., when mocking). It also creates incorrect results when a domain name is resolved to the same IP address. e.g.,

        URL u1 = new URL("http://sourceforge.net/");
        URL u2 = new URL("http://ch3.sourceforge.net/");
        System.out.println(u1.sameFile(u2));

will print true since ch3.sourceforge.net and sourceforge.net resolves to the same IP address (it does for me at least, YMMV). That does not mean that the web server will serve the same file for both Host headers (e.g., when having multiple virtual hosts served by the same HTTP server), and I don't believe this is correct behavior for WebClient.

I believe HtmlUnit should not use java.net.URL#sameFile. Maybe by using org.apache.commons.httpclient#URI, or by comparing the individual fields of the URL object (mapping "" -> "/" for empty paths, doing IDNA ToASCII comparisons for names, and all the other things that are needed).

//Sebastian Cato

[RBRi]

Hi Sebastion,
have changed the impl to have our own impl of 'sameFile' that no longer does the DNS lookup and add some path normalization.
Please check if this cures your problems.

[Sebastian Cato]

Hello,

After checking out r12709 of trunk and building, I see a reduction of the number of DNS requests by ~41% for the site I'm testing against (internal).

There were other redundant DNS requests being issued which I attributed to the use of java.net.URL#samefile before. I tracked these down to something that's not a part of HtmlUnit, so I'm happy with the fix.

However in the process I learned that java.net.URL#equals is the same as java.net.URL#sameFile, so when comparing two URL objects one may cause a DNS request to be performed. This is done in com.gargoylesoftware.htmlunit.util.UrlUtils#areSame (UrlUtils.java:998, right above the newly added sameFile). This was not a source of the DNS requests I saw though, but I figure it's worth mentioning.

Thanks!

[Ahmed Ashour]

Thanks a lot for your feedback. UrlUtils was changed, not to use URL.equals().