httpunit / Bugs / #347 Downloading gif images uses up sockets

Sir Runcible Spoon - 2006-02-16

Logged In: YES
user_id=1122348

I've found the cause.

WebResponse.loadResponseText() is only called when the
ContentType begins with "text" (say "text/html",
"text/javascript"). It is loadResponseText() that reads the
previously opened input stream, parses the tags, and closes
it. When lots of responses of other types are received we
build up a load of unclosed streams and run out of file
handles (I guess).

I have patched my version of WebResponse.java and
HttpWebResponse.java to use another method ignoreResponse()
for non-text responses. This simply reads the stream and
closes it.

See attached sources. This is just a hack that works around
the problem.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sir Runcible Spoon - 2006-02-16

Workaround sources

patchedSources.zip

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Wolfgang Fahl - 2008-04-19

assigned_to: nobody --> wolfgang_fahl

status: open --> pending-accepted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Wolfgang Fahl - 2008-04-19

Logged In: YES
user_id=1220573
Originator: NO

Dear Sir Runcible Spoon,

thank you for your bug report. I tried to reproduce your problem with:
http://httpunit.svn.sourceforge.net/viewvc/httpunit?view=rev&revision=935

but was not able to. With many images the tests take very long - so a first indicator of your patch working would be that the tests run much faster.
Your patch is against some code of 2006 - I'm sorry that it took so long to react and therefore we'd need to get diffs againgst the latest trunk revision to look into this. I'd appreciate your contribution for this.

I'm looking forward to your coooperation.

Yours
Wolfgang

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Wolfgang Fahl - 2008-04-19

Logged In: YES
user_id=1220573
Originator: NO

Need more information to do anything about this. What exactly
doesn't work?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sir Runcible Spoon - 2008-04-19

Logged In: YES
user_id=1122348
Originator: YES

My, this was a long time ago.

Having looked at the available downloads. Diffing the sources, the sources I pinned up must have been for 1.6. However the problem still existed in 1.6.1 and 1.6.2, but I haven been able to work out how to build the current trunk. They look very different, so I guess it has been greatly re-worked.

If you would like to test it post a 1x1.gif on your localhost and use the following program:

import com.meterware.httpunit.*;

public class GifDownLoader
{
public static void main(String[] args) throws Exception
{
WebConversation session = new WebConversation();
int i = 1;
while(true)
{
System.out.println(i++);
WebResponse image = session.getResponse("http://localhost/1x1.gif");
}
}
}

It packs up at this point on XP:

...
3964
3965
3966
3967
Exception in thread "main" java.net.BindException: Address already in use: connect
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
...

Cheers

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sir Runcible Spoon - 2008-04-19

status: pending-accepted --> open-accepted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Roger Lindsjö - 2008-04-21

Logged In: YES
user_id=643287
Originator: NO

Isn't this just due to Windows by default limiting the number of sockets? Short explanation, you can not reuse a socket withing 2 minutes, and by default Windows will only allow you ot use sockets 1024-5000, so if you create more than 33 sockets per seconds you will run in to this problem.

See: http://forums.parasoft.com/index.php?showtopic=435

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sir Runcible Spoon - 2008-04-22

Logged In: YES
user_id=1122348
Originator: YES

No. The problem is that the socket used is not closed. If you change the url in my example program to download a HTML file (say http://localhost/index.html\) then httpunit will close the socket after each request and continue to download the file indefinately without problems.

My changes to 1.6 make httpunit deal file types that it does not recognise. It simply downloads the whole response from the already open socket (which it then ignores) and then it closes the socket.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Russell Gold - 2008-04-24

Logged In: YES
user_id=37920
Originator: NO

The proposed fix breaks the use of a web response to obtain a binary stream; it appears that what we need is a pool of open webresponse objects. Once the stream is closed, we remove the item from the pool. If the pool becomes too large (settable), we close the input stream of the oldest entry.

To test this, we need a way emulate socket assignments, and I don't have an answer off the top of my head for how to do that.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Roger Lindsjö - 2008-04-24

Logged In: YES
user_id=643287
Originator: NO

I tried the following in WebClientTest, does that replicate the error?

public void testManyImages() throws Exception {
WebConversation wc = new WebConversation();
defineResource( "image", new byte[1024], "image/gif");
for (int i = 0; i < 10000; i++) {
WebResponse resp = wc.getResponse(getHostPath() + "/image");
// Enable line below to not get too many open files / sockets
//resp.getInputStream().close();
}
}

I'm not sure why the input stream ins not always read, perhaps in order to handle streaming (large) content. Anyway, with the change where I get the input stream and explicitly close it, then I don't get the error. Reading the code this seems clear, but since the close is not needed for text/* content, this should be fixed or documented (unless all users are supposed to read and understand the code for httpunit).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Wolfgang Fahl - 2008-04-29

Logged In: YES
user_id=1220573
Originator: NO

Thank you for your further input.

I have added your test case proposals and tried out the results on an XP machine. I get some 3889, over 5700 and even 15000 tries as varying results.
Very strange ... The test cases are commented out for the current HttpUnitSuite until we clarify what to do.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Wolfgang Fahl - 2008-04-29

status: open-accepted --> pending-accepted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Roger Lindsjö - 2008-05-01

Logged In: YES
user_id=643287
Originator: NO

One possible reason for getting varying number could be the because of the garbage collection. Since the test did not hold on to the WebResponse, it was ellageable for gc, which in turn also frees the stream. For the streams I think there is a finally clause which closes the them. So, with little memory GC runs more often, thus actually closing the streams. With much memory it does not. On my machine the default setting is actually 512MB, so I don't think any gc occurs during the test.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

SourceForge Robot - 2008-05-14

Logged In: YES
user_id=1312539
Originator: NO

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

SourceForge Robot - 2008-05-14

status: pending-accepted --> closed-accepted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Downloading gif images uses up sockets

Group

Searches

Help

#347 Downloading gif images uses up sockets

Discussion