#1679 Memory Leak because of ThreadLocal in HttpWebConnection

[Carsten]

If you have long running threads, for example a thread pool, with more than one thread using a single WebClient instance it will not be garbage collected because the ThreadLocalMap of every thread using it has an indirect reference to it.

For testing purposes I reverted the changes from http://sourceforge.net/p/htmlunit/code/10314/ and the memory leak disappeared. A better solution would be to use RequestBuilder in makeHttpMethod() instead of reconfiguring the same HttpClientBuilder and building a new HttpClient for every request.

Below is an export from Eclipse Memory Analyzer of the heap from the attached test.

~~~~~~~~~~~~~~~~

Class Name | Shallow Heap | Retained Heap

                                                                                                                                                                     |              |

com.gargoylesoftware.htmlunit.WebClient @ 0x64a67de80 | 144 | 22.592
'- webClient_ com.gargoylesoftware.htmlunit.HttpWebConnection @ 0x64a67de48 | 40 | 25.528
|- this$0 com.gargoylesoftware.htmlunit.HttpWebConnection$3 @ 0x64a6aa6d8 | 24 | 24
| '- redirectStrategy org.apache.http.impl.client.HttpClientBuilder @ 0x64a663d48 | 200 | 1.048
| '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x64a663d28 | 32 | 1.080
| '- [5] java.lang.ThreadLocal$ThreadLocalMap$Entry[16] @ 0x64a6623a0 | 80 | 3.288
| '- table java.lang.ThreadLocal$ThreadLocalMap @ 0x64a662388 | 24 | 3.312
| '- threadLocals com.gargoylesoftware.htmlunit.HttpWebConnectionMemoryLeakTest$RequestHandler @ 0x64a610588 Thread-0 Busy Monitor, Native Stack, Thread| 128 | 3.664
|- this$0 com.gargoylesoftware.htmlunit.HttpWebConnection$3 @ 0x64a6ab9d8 | 24 | 24
| '- redirectStrategy org.apache.http.impl.client.HttpClientBuilder @ 0x64a6ab208 | 200 | 2.328
| '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x64a6ab1e8 | 32 | 2.360
| '- [5] java.lang.ThreadLocal$ThreadLocalMap$Entry[16] @ 0x64a6aa950 | 80 | 4.560
| '- table java.lang.ThreadLocal$ThreadLocalMap @ 0x64a6aa938 | 24 | 4.584
| '- threadLocals com.gargoylesoftware.htmlunit.HttpWebConnectionMemoryLeakTest$RequestHandler @ 0x64a610408 Thread-1 Busy Monitor, Native Stack, Thread| 128 | 4.832
|- this$0 com.gargoylesoftware.htmlunit.HttpWebConnection$3 @ 0x64a6c9818 | 24 | 24
| '- redirectStrategy org.apache.http.impl.client.HttpClientBuilder @ 0x64a6c9048 | 200 | 2.328
| '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x64a6c9028 | 32 | 2.360
| '- [5] java.lang.ThreadLocal$ThreadLocalMap$Entry[16] @ 0x64a6c8e20 | 80 | 2.880
| '- table java.lang.ThreadLocal$ThreadLocalMap @ 0x64a6c8e08 | 24 | 2.904
| '- threadLocals com.gargoylesoftware.htmlunit.HttpWebConnectionMemoryLeakTest$RequestHandler @ 0x64a618358 Thread-3 Busy Monitor, Native Stack, Thread| 128 | 3.152
|- this$0 com.gargoylesoftware.htmlunit.HttpWebConnection$3 @ 0x64a723148 | 24 | 24
| '- redirectStrategy org.apache.http.impl.client.HttpClientBuilder @ 0x64a67cd78 | 200 | 2.328
| '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x64a67cd58 | 32 | 2.360
| '- [5] java.lang.ThreadLocal$ThreadLocalMap$Entry[16] @ 0x64a67cb50 | 80 | 2.880
| '- table java.lang.ThreadLocal$ThreadLocalMap @ 0x64a67cb38 | 24 | 2.904
| '- threadLocals com.gargoylesoftware.htmlunit.HttpWebConnectionMemoryLeakTest$RequestHandler @ 0x64a6189e8 Thread-4 Busy Monitor, Native Stack, Thread| 128 | 3.152
|- this$0 com.gargoylesoftware.htmlunit.HttpWebConnection$3 @ 0x64a7bde40 | 24 | 24
| '- redirectStrategy org.apache.http.impl.client.HttpClientBuilder @ 0x64a7957b8 | 200 | 2.328
| '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x64a795798 | 32 | 2.360
| '- [5] java.lang.ThreadLocal$ThreadLocalMap$Entry[16] @ 0x64a795590 | 80 | 2.880
| '- table java.lang.ThreadLocal$ThreadLocalMap @ 0x64a795578 | 24 | 2.904
| '- threadLocals com.gargoylesoftware.htmlunit.HttpWebConnectionMemoryLeakTest$RequestHandler @ 0x64a608d80 Thread-2 Busy Monitor, Native Stack, Thread| 128 | 3.152
'- Total: 5 entries | |

---

~~~~~~~~~~~~~~~

[RBRi]

I'm working on that already but i fear this will require some time to clean up all (or at least some) of the hacks from the past until we are more or less in sync with HttpClient.
So please be patient or provide patches ;-)

[RBRi]

Can you please have a quick look, i guess the reference to the web client is gone with the latest code.

[Carsten]

That reference is gone but there is another one.
It seems like the extending of StaticHttpRequestInterceptor in getHttpRequestInterceptors() is the culprit.

Class Name                                                                                                                                                                     | Shallow Heap | Retained Heap
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
com.gargoylesoftware.htmlunit.WebClient @ 0x64a6686a0                                                                                                                          |          144 |        23.560
'- webClient_ com.gargoylesoftware.htmlunit.HttpWebConnection @ 0x64a668668                                                                                                    |           40 |        49.408
   |- this$0 com.gargoylesoftware.htmlunit.HttpWebConnection$9 @ 0x64a668580                                                                                                   |           24 |            24
   |  '- [0] org.apache.http.HttpRequestInterceptor[10] @ 0x64a6681e8                                                                                                          |           56 |           432
   |     '- requestInterceptors org.apache.http.protocol.ImmutableHttpProcessor @ 0x64a6681d0                                                                                  |           24 |           776
   |        '- httpprocessor org.apache.http.impl.client.HttpClientBuilder @ 0x64a667160                                                                                       |          200 |         2.328
   |           '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x64a667140                                                                                               |           32 |         2.360
   |              '- [5] java.lang.ThreadLocal$ThreadLocalMap$Entry[16] @ 0x64a666f38                                                                                          |           80 |         2.880
   |                 '- table java.lang.ThreadLocal$ThreadLocalMap @ 0x64a666f20                                                                                               |           24 |         2.904
   |                    '- threadLocals com.gargoylesoftware.htmlunit.HttpWebConnectionMemoryLeakTest$RequestHandler @ 0x64a61b528  Thread-4 Busy Monitor, Native Stack, Thread|          128 |         3.152
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[RBRi]

Yep, you are right. Have simplified the code again, hope we are getting closer....

[RBRi]

Sorry for the confusion, but the HttpClient API is a bit strange. Hope the last 'hack' cures the problem (at least the Interceptor).
Waiting for your feedback....

[Carsten]

The WebClient reference is gone. But as ThreadLocal is still used a lot of objects that are not garbage collected remain.

~~~~~~~~~~~~~~
Object / Stack Frame |Name | Shallow Heap | Retained Heap |

---

com.gargoylesoftware.htmlunit.HttpWebConnectionMemoryLeakTest$RequestHandler @ 0x64a60bb50 |Thread-0| 128 | 3.864 |
...
| | |- threadLocals java.lang.ThreadLocal$ThreadLocalMap @ 0x64a6c60f8 | | 24 | 3.512 |
| | | |- table java.lang.ThreadLocal$ThreadLocalMap$Entry[16] @ 0x64a6c6110 | | 80 | 3.488 |
| | | | |- [5] java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x64a6d5e28 | | 32 | 1.280 |
| | | | | |- value org.apache.http.impl.client.HttpClientBuilder @ 0x64a6d5e48 | | 200 | 1.248 |
| | | | | | |- <class> class org.apache.http.impl.client.HttpClientBuilder @ 0x64a6ee1b0 | | 0 | 0 |
| | | | | | |- connManager org.apache.http.impl.conn.PoolingHttpClientConnectionManager @ 0x64a680fc8 | | 32 | 1.040 |
| | | | | | |- defaultSocketConfig org.apache.http.config.SocketConfig @ 0x64a6810a0 | | 40 | 40 |
| | | | | | |- connTimeToLiveTimeUnit java.util.concurrent.TimeUnit$3 @ 0x64a6814f8 | | 24 | 88 |
| | | | | | |- sslSocketFactory com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory @ 0x64a681680 | | 40 | 1.032 |
| | | | | | |- httpprocessor org.apache.http.protocol.ImmutableHttpProcessor @ 0x64a6d5f10 | | 24 | 768 |
| | | | | | |- redirectStrategy com.gargoylesoftware.htmlunit.httpclient.HtmlUnitRedirectStrategie @ 0x64a6d6210 | | 16 | 16 |
| | | | | | |- cookieSpecRegistry org.apache.http.config.Registry @ 0x64a6d6220 | | 16 | 192 |
| | | | | | |- cookieStore com.gargoylesoftware.htmlunit.httpclient.HtmlUnitCookieStore @ 0x64a6d62e0 | | 16 | 16 |
| | | | | | |- credentialsProvider com.gargoylesoftware.htmlunit.DefaultCredentialsProvider @ 0x64a6d6c98 | | 16 | 64 |
| | | | | | |- defaultRequestConfig org.apache.http.client.config.RequestConfig @ 0x64a6d6cd8 | | 56 | 56 |</class>

---

~~~~~~~~~~~~~~~~