Re: [Htmlunit-user] How to get WebResponse for HtmlPage resources

[Aaron B. <Aar...@ad...>]

Thanks for the response, but turns out there IS a way to do this, that’s actually officially supported!

After much more searching and seeking, I found this FAQ http://htmlunit.sourceforge.net/faq.html#HowToModifyRequestOrResponse. So simply using the WebConnectionWrapper, I implemented some WebResponse caching.

For those interested, here is the code I used. Probably can be cleaned up a bit more, but it works just fine. Just be sure to use the stopCaching() which will clear the cache, otherwise you run into danger of running out of memory after extended usage.

//Setting the object on the client
webClient.setWebConnection(new RequestCachingWebConnection(ret.getWebConnection()));


//make a call to a web page while caching all of the WebRequests that are made, including loading resources on the web page
((RequestCachingWebConnection)webClient.getWebConnection()).startCaching();
page = webClient.getPage(url);
List<WebResponse> responses = ((RequestCachingWebConnection)webClient.getWebConnection()).stopCaching();
for( WebResponse response : responses ) {
    //do stuff
}


public class RequestCachingWebConnection extends WebConnectionWrapper {
    private final List<WebResponse> responses = new CopyOnWriteArrayList<>();
    private boolean doCache = false;

    RequestCachingWebConnection(WebClient webClient) {
        super(webClient);
    }

    RequestCachingWebConnection(WebConnection webConnection) {
        super(webConnection);
    }

    @Override
    public WebResponse getResponse(WebRequest request) throws IOException {
        WebResponse response = super.getResponse(request);
        if( doCache ) {
            responses.add(response);
        }
        return response;
    }

    public void startCaching() {
        responses.clear();
        doCache = true;
    }

    public List<WebResponse> stopCaching() {
        if( doCache ) {
            doCache = false;
            List<WebResponse> tmp = new ArrayList<>(responses);
            responses.clear();
            return tmp;
        }
        return new ArrayList<>();
    }

    public List<WebResponse> getCurrentResponses() {
        return new ArrayList<>(responses);
    }
}

Aaron Baff
Lead Java Developer
e: aar...@ad...<mailto:aar...@ad...>
t:  310.556.4440 x29
f:  310. 556. 4441


[cid:6E2...@wa...]

On Jan 17, 2016, at 7:24 AM, Ronald Brill <rb...@rb...<mailto:rb...@rb...>> wrote:

On Fri, 15 Jan 2016 12:11:01 -0800 Aaron Baff wrote:

I'm trying to figure out how to get the WebResponse (or similar type of data) for resources found on an HtmlPage. Specifically in this case, I'd like to get the HTTP Headers returned by a <script> tag with a src to a js file on the server. Is there any
way to do this?


Hi Aaron,

i fear, there is no way to do it at the moment by using the HtmlUnit API. Internally the WebResponse is thrown away at the moment the string content was created from the response.
If you only like to see the headers, you can enable wire logging for HttpClient. That is simple and you can see everything send out or received from HtmlUnit.
Or if you need a bit more convenience try to work with some WebProxy like Charles or Fiddler.

Hope this helps

   RBRi
   --------------------------
   WETATOR
   Smart web application testing
   http://www.wetator.org



Aaron Baff
Lead Java Developer
e: aar...@ad...<mailto:aar...@ad...><mailto:aar...@ad...>
t:  310.556.4440 x29
f:  310. 556. 4441


[cid:6E2...@wa...]


________________________________
CONFIDENTIALITY NOTICE
This communication (and/or the documents accompanying it) may contain confidential information. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on
the
contents of this information is strictly prohibited. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents.

<<< Inline attachment: image001.png - [image/png] >>>


----< Inline text [text-plain-05.txt] >------------------

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140


----< Inline text [text-plain-06.txt] >------------------

_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user





________________________________
CONFIDENTIALITY NOTICE
This communication (and/or the documents accompanying it) may contain confidential information. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents.