Re: [Springframework-developer] Cleanup of context resources on webapp reload

[Tim K. <tim...@vi...>]

Jurgen,

Thanks for looking into this issue.  When I originally reported on it, =20=

and didnt hear much discussion on it, I had started to think that i was =20=

the only person experiencing the memory leak and that I was coding =20
something wrong.

Your experiences w/ JProfiler are very similar to what I was seeing =20
when I was running JProfiler against our code, although you've =20
obviously delved deeper into the exact reasons for those leaks.  I will =20=

give the latest build of spring a try on our code and hopefully it =20
should help alleviate the situation some.

I also wanted to add a comment in respect to what Guillaume said below =20=

that we've consistently seen OOM errors coming from Tomcat after X =20
amount of reloads done by the test server.  And this can happen over =20
the span of several days, so I do not believe that those resources are =20=

going to be GC'ed at all, unless the definition of "took a while" means =20=

a week or so.  :)   Resin may be doing something different however.

-tim


On May 24, 2004, at 8:48 AM, Guillaume Poirier wrote:

> Well, I find it really weird, because I've tested such behavior before =
=20
> on
> Resin 2.1.10, and while it took a while before the resources were =20
> garbage
> collected, they were always eventually collected.  And if I added a
> System.gc() when the ServletContext was loaded, all my singletons were =
=20
> being
> collected right away.  During all your tests, did you notice higher =20=

> memory
> consumption or did you actually get a OutOfMemoryError ?  May be it's =20=

> the
> profiler that keeps a reference on some resource(s) for some reasons?
>
> But if it is really a ClassLoader issue, shouldn't you be able to =20
> simulate
> the situation outside a Servlet Container?  I've run the attached =20
> test, and
> while I create a memory leak quickly if I keep a reference on the
> ClassLoader, I am not able to produce one without holding a ref on it.
>
> Or is my test flawed?
>
> Guillaume
>
> ----- Original Message -----
> From: "j=FCrgen h=F6ller [werk3AT]" <jue...@we...>
> To: <spr...@li...>
> Sent: Monday, May 24, 2004 4:41 AM
> Subject: Re: [Springframework-developer] Cleanup of context resources =20=

> on
> webapp reload
>
>
> Just finished tests on Resin 2.1.11 - exactly same behavior as with =20=

> Tomcat.
> This *is* a general class loader respectively garbage collection =
issue,
> rather than a server-specific leak.
>
> Consequently, with the changes I've committed yesterday, there is the =20=

> same
> significant benefit as with Tomcat. The only remaining issue are the =20=

> "full
> object" constants like ClassFilter.TRUE.
>
> Juergen
>
>
> ________________________________
>
> Von: spr...@li... im Auftrag =20=

> von
> j=FCrgen h=F6ller [werk3AT]
> Gesendet: Mo 24.05.2004 09:20
> An: spr...@li...
> Betreff: Re: [Springframework-developer] Cleanup of context resources =20=

> on
> webapp reload
>
>
>
> FYI, I've tested both Tomcat 5.0.18 and 4.1.27 - same behavior with =20=

> both.
>
> BTW, a couple of related posts from the Resin mailing list:
>
> http://www.caucho.com/support/resin-interest/0404/0117.html
> http://www.caucho.com/support/resin-interest/0111/0164.html
> <http://www.caucho.com/support/resin-interest/0111/0164.html>
>
> Juergen
>
>
> ________________________________
>
> Von: spr...@li... im Auftrag =20=

> von
> j=FCrgen h=F6ller [werk3AT]
> Gesendet: Mo 24.05.2004 08:42
> An: spr...@li...
> Betreff: Re: [Springframework-developer] Cleanup of context resources =20=

> on
> webapp reload
>
>
>
> I've told JProfiler to explicitly run garbage collection - a number of
> times, actually - before I've had a look at the heap. So I'm sure that =
=20
> those
> remaining objects were not garbage-collected, and probably would have =20=

> stayed
> around in the VM forever...
>
> I'm aware that it seems odd, but this issue just seems to affect =20
> specific
> static fields: BeanWrapperImpl's defaultEditors did not cause a leak, =20=

> but
> CachedIntrospectionResults' classCache did. Normal constants or static
> logger fields didn't, but "full object" constants like =20
> ClassFilters.TRUE
> did.
>
> CachedIntrospectionResults' classCache uses the Class as key; the =20
> value, a
> CachedIntrospectionResult object, also refers to the key Class.
> Consequently, I had to use a WeakHashMap *with WeakReferences as =20
> values* to
> see proper garbage collection. Note that this static cache contains
> instances of its containing class as values.
>
> I've run all my tests x times to make sure that I could trust my eyes. =
=20
> I've
> even undone the WeakReference changes again, and voila, there were the =
=20
> leaks
> again. I'd be happy to learn more about how the garbage collector =
works
> here... All I can state at this point of time is that the changes did =20=

> cause
> an obvious difference in terms of resource leaks.
>
> Juergen
>
>
> ________________________________
>
> Von: spr...@li... im Auftrag =20=

> von
> Guillaume Poirier
> Gesendet: Mo 24.05.2004 05:17
> An: spr...@li...
> Betreff: Re: [Springframework-developer] Cleanup of context resources =20=

> on
> webapp reload
>
>
>
> The "resource leak" caused by a singleton when a webapp's classloader =20=

> is
> thrown away is only temorary, the unused classes and the classloader =20=

> will be
> eventually garbage collected and the resources will be freed.  The =
only
> thing that could prevent that is if there was a something in the =20
> server's
> classloader that still had a reference on an object or a class of the =20=

> child
> classloader.
>
> Unless there's a bug in Tomcat or in the application code, I really =20=

> can't
> see how the classes won't be eventually garbage collected when the JVM =
=20
> needs
> memory.  And anyway, if there was indeed a leak because =20
> SQLErrorCodesFactory
> is a singleton, why wouldn't there be one for each static fields such =20=

> as
> constants?
>
> Are you sure that JProfiler does not disable garbage collecting in =20
> order to
> make its profiling?  I know that in many of the JVMPI method calls are =
=20
> done
> with garbage collecting off.  I suspect the above to be the cause of =20=

> the
> "resource leak", rather than any singleton that Spring might be using.
>
> Guillaume
>
> ----- Original Message -----
> From: "j=FCrgen h=F6ller [werk3AT]" <jue...@we...>
> To: <spr...@li...>
> Sent: Sunday, May 23, 2004 4:45 PM
> Subject: Re: [Springframework-developer] Cleanup of context resources =20=

> on
> webapp reload
>
>
> I've just spent about 10 hours profiling Spring, using the Image =20
> Database
> and Petclinic samples. (BTW, I've used an evaluation version of =20
> JProfiler
> from ej-technologies - nice product!)
>
> Although I still don't completely understand the garbage collection
> behavior, I've figured out the following issues. Each of them simply
> prevents the respective classes from getting garbage collected on
> destruction of the class loader (e.g. on Tomcat web app shutdown).
>
> - A classic singleton with a class variable holding the object. I've
> reworked GlobalAdvisorAdapterRegistry and SQLErrorCodesFactory to hold =
=20
> the
> respective singleton as a WeakReference.
>
> - A static cache. I've reworked CachedIntrospectionResults to use a
> WeakHashMap with WeakReferences as values.
>
> - A ThreadLocal with a default other than null. I've reworked
> TransactionSynchronizationManager to use null as default for the =20
> resource
> map, setting a HashMap there on demand, removing the entire HashMap =20=

> when
> unbinding the last resource.
>
> - Constants that define a full object. We have a number of those, for
> example ClassFilters.TRUE and AdvisedSupport.EMPTY_TARGET_SOURCE. I've =
=20
> tried
> for quite a while, but I haven't been able to figure out a way to =20
> define
> such constants such that they will be garbage collected.
>
> The latter programming style is not uncommon, so I really don't =20
> understand
> why it causes trouble with garbage collection. Hibernate uses a =
similar
> style for its FlushMode, for example.
>
> In general, other frameworks like CGLIB, Hibernate, Velocity have huge
> resource leaks on web app shutdown, while just the constants issue =20
> remains
> with Spring now. As long as those huge third-party leaks are not =20
> addressed,
> I'm not worried at all by the single remaining Spring issue.
>
> As I initially said, we shouldn't exaggerate the problem, as it =20
> basically
> just affects hot reloading of web apps - mainly a development feature
> anyway. We need to make that clear to users too, to avoid comments a =
la
> "Spring is not usable for real apps because it leaks on hot =20
> redeployment".
>
> Please, everybody, give the current CVS head a sanity check tomorrow. =20=

> There
> shouldn't be any issues: the test suite passes, the sample apps run
> properly. Still, I'd feel more comfortable if we make sure that no =20
> subtle
> side effects have been introduced.
>
> For this reason, I will delay release 1.0.2 till tomorrow night.
>
> Juergen
>
>
> ________________________________
>
> Von: spr...@li... im Auftrag =20=

> von
> j=FCrgen h=F6ller [werk3AT]
> Gesendet: Sa 22.05.2004 15:52
> An: spr...@li...
> Betreff: Re: [Springframework-developer] Cleanup of context resources =20=

> on
> webapp reload
>
>
>
> So basically, all static caches cause resource leaks when restarting a
> Tomcat web app? I wonder why this happens... The class loader should
> completely dissolve all classes that it has loaded in its lifetime,
> including static caches. Or have I misunderstood something here? =
Anyone
> having in-detail experience with handling such a scenario?
>
> Juergen
>
>
> ________________________________
>
> Von: spr...@li... im Auftrag =20=

> von
> Dmitriy Kopylenko
> Gesendet: Di 04.05.2004 18:02
> An: spr...@li...
> Betreff: Re: [Springframework-developer] Cleanup of context resources =20=

> on
> webapp reload
>
>
>
> Well,
>
> for instance SQLErrorCodesFactory is a singleton(GoF, not Spring) =
which
> caches SQLErrorCodes internally in the Map with strong references. =20
> Again, I
> don't know if trying
> to use WeakHashMap there would do the trick...
>
> Dmitriy
>
> Tim Kettering wrote:
>
>>
>> I looked at it some more this morning, and basically what I did was
>> start up tomcat w/ the webapp in the profiler, then after it was done
>> starting up I used tomcat's manager to stop the context.  This should
>> destroy all resources related to the context.   Here is a list of
>> spring related stuff that still were in memory after the context was
>> closed.  Other stuff was cleaned up just fine.
>>
>> org.springframework.beans.CachedIntrospectionResults
>> org.springframework.jdbc.support.SQLCodes
>> org.springframework.core.Constants
>> org.springframework.aop.framework.AdvisedSupport$1
>> org.springframework.aop.framework.adapter.BeforeAdviceAdapter
>> org.springframework.aop.framework.adapter.AfterReturningAdviceAdapter
>> org.springframework.jdbc.support.SQLErrorCodesFactory
>> =
org.springframework.transaction.support.TransactionSynchronizationMana=20=

>> ge
>> r$1
>> org.springframework.transaction.interceptor.RollbackRuleAttribute
>> org.springframework.aop.framework.adapter.ThrosAdviceAdapter
>> org.springframework.aop.Pointcut$1
>> =
org.springframework.aop.framework.adapter.GlobalAdvisorAdapterRegistry
>>
>> On May 3, 2004, at 3:28 PM, Dmitriy Kopylenko wrote:
>>
>>> I'm just wondering, would the use of WeakHashMap in
>>> CachedIntrospectionResults help?
>>>
>>> Dmitriy.
>>>
>>> Tim Kettering wrote:
>>>
>>>> I posted this to the users list last week and did not receive any
>>>> reply  on it, so I'm posting it again here on the developer list, =
in
>>>> hopes i  could get an reply from someone here.   I'm trying to
>>>> determine if its  something I should be doing myself, or if hte
>>>> spring context should be  cleaning up those resources by itself on
>>>> the .close() call.  Further  profiling shows that there are
>>>> duplicate  instances of SQLError and  hibernate proxy classes =20
>>>> hanging
>>>> around  afterwards too.  Other objects  do get cleaned up properly.
>>>> --------
>>>> Hi everyone,
>>>> We're (meaning me) looking into some resource leaks that are
>>>> occuring   when our webapp context gets reloaded.  I found that
>>>> context.close()   needs to be called on the destroy() method of
>>>> plugin we're using,  and  it works for a good majority of the =20
>>>> objects
>>>> we were seeing  leaked, but  there are some objects that I'm unable
>>>> to make go away.  Object in  question is the:
>>>> org.springframework.beans.CachedIntrospectionResults
>>>> Whenever I reload the context - the profiler I'm using shows that I
>>>> have essentially a duplicate group of those objects (same instance
>>>> count) as the original, and successive reloads will continue to
>>>> duplicate this.
>>>> The profiler also shows the final reference to those objects like  =20=

>>>> this:
>>>> 100% - 1008 bytes - 63 alloc.
>>>> =
org.springframework.context.support.ClassPathXmlApplicationContext.<=20
>>>> in
>>>> it >
>>>> So basically I guess what I'm asking is for ideas or suggestions on
>>>> how  I could get those to clean up.  This bean doesnt show up in =
the
>>>> Spring  javadocs.  And looking in CVS says its a package level =
bean,
>>>> not for  application use, so I'm thinking that closing the context
>>>> should (in  theory) clean this up?   Thanks in advance.
>>>> -tim
>>>> -------------------------------------------------------
>>>> This SF.Net email is sponsored by: Oracle 10g
>>>> Get certified on the hottest thing ever to hit the market... Oracle
>>>> 10g. Take an Oracle 10g class now, and we'll give you the exam =
FREE.
>>>> http://ads.osdn.com/?ad_id=3D3149&alloc_id=3D8166&op=3Dclick
>>>> _______________________________________________
>>>> Springframework-developer mailing list
>>>> Spr...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/springframework-=20
>>>> developer
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> This SF.Net email is sponsored by: Oracle 10g
>>> Get certified on the hottest thing ever to hit the market... Oracle
>>> 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
>>> http://ads.osdn.com/?ad_id=3D3149&alloc_id=3D8166&op=3Dclick
>>> _______________________________________________
>>> Springframework-developer mailing list
>>> Spr...@li...
>>> https://lists.sourceforge.net/lists/listinfo/springframework-=20
>>> developer
>>>
>>
>>
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by: Oracle 10g
>> Get certified on the hottest thing ever to hit the market... Oracle =20=

>> 10g.
>> Take an Oracle 10g class now, and we'll give you the exam FREE.
>> http://ads.osdn.com/?ad_id=3D3149&alloc_id=3D8166&op=3Dclick
>> _______________________________________________
>> Springframework-developer mailing list
>> Spr...@li...
>> =
https://lists.sourceforge.net/lists/listinfo/springframework-developer
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle =20=

> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3D3149&alloc_id=3D8166&op=3Dclick
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle =20=

> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id149&alloc_id=9966&op=3Dick
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle =20=

> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id149&alloc_id=9966&op=3Dick
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle =20=

> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3D3149&alloc_id=3D8166&op=3Dclick
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle =20=

> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id149&alloc_id=9966&op=3Dick
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle =20=

> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id149&alloc_id=9966&op=3Dick
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle =20=

> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id149&alloc_id=9966&op=3Dick
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
> <classloader.tar.gz>=