I've been trying out Arachnid this morning and I may have found a little glitch:
"boolean isExternalSite(URL link)"
of the Arachnid class compares
"link.getAuthority()" to "baseUrl.getAuthority()" with a "!=" operator.
Both of these are strings, though.
IMHO this line should read something like
(Please do correct me if I'm mistaken.)
Secondly a suggestion:
there should be a way to specify a proxy, through which connections are to be opened.
(It works if you put the following lines before the code that opens the HttpURLConnection, i.e. at the very beginning of the getWebPage method:
It would be cleaner, though, if you were able to pass it to the method somehow..