From: SourceForge.net <no...@so...> - 2004-08-02 07:53:25
|
Bugs item #435958, was opened at 2001-06-25 01:53 Message generated for change (Comment added) made by cazzius You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=376685&aid=435958&group_id=22866 Category: JBossServer Group: v3.2 Status: Closed Resolution: Fixed Priority: 5 Submitted By: Nobody/Anonymous (nobody) Assigned to: Claudio Vesco (cazzius) Summary: Out of Memory eception after many redepl Initial Comment: This behavior has been observed on Windows 2000 SP2, running Sun JDK 1.3.1 and jBoss2.2.2 both Tomcat and Jetty bundles. After many EAR hot redeployments in a short period of time, (prehaps one every 5 to 20 minutes over a period of 6 hours) a redeployment will result in "Out of Memory" exceptions being thrown. This happens with both the Tomcat and Jetty bundles, but the error is much more clear with Jetty. In the case of Tomcat, it appears to redeploy correctly but creates a variety of errors when the app is run. ---------------------------------------------------------------------- >Comment By: Claudio Vesco (cazzius) Date: 2004-08-02 07:53 Message: Logged In: YES user_id=211618 I think there are still some problems... When you deploy/undeploy an application on Windows, some times jboss cannot delete the local copy. This is caused by a classloader which is not gc because there is a strong reference to it. I have tried to determine the strong references. I think that there are strong references on: 1) JarURLConnection :-( 2) security cache: LoginContext hold a strong reference to the jaas module classloader LoginContext.contextClassLoader On my local cvs repository (Branch 3.2) I have resolved these issues :-) but I need some more tests :-( For the first problem, I have forced defaultUseCaches to false on an instance of JarURLConnection in org.jboss.system.server.ServerImpl. The second problem is a little more complex :-( When there are no more classloader references (no java.lang.ref reference :-) ), I trigger an event. This event invalidate the security cache entry which is associated with the classloader. Sorry for my english. ---------------------------------------------------------------------- Comment By: Scott M Stark (starksm) Date: 2004-08-01 13:53 Message: Logged In: YES user_id=175228 Every specific example that has been provided has been fixed. Open a new case with example deployment which causes the problem. ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-08-01 11:42 Message: Logged In: YES user_id=345880 It's been a three-year anniversary of this bug recently. JBoss versions continue to grow, I have installed 3.2.5 and still need to monitor memory and periodically restart our production servers. I consider this bug is the only one that makes JBoss not a production quality product. Is there any estimation when it may be fixed? Thanks. ---------------------------------------------------------------------- Comment By: Claudio Vesco (cazzius) Date: 2003-12-17 08:06 Message: Logged In: YES user_id=211618 I have found some memory leak caused by the use of JarURLConnection and jboss jmx. I'll try to kill this very old bug shortly. ---------------------------------------------------------------------- Comment By: Tim McCune (javajedi) Date: 2003-12-11 19:35 Message: Logged In: YES user_id=62441 This is now the oldest JBoss bug (2.5 years). I'm assuming that means it's going to be damn hard to fix. :) Anyone care to comment on any progress being made here? ---------------------------------------------------------------------- Comment By: IM (imaniuk) Date: 2003-12-03 20:16 Message: Logged In: YES user_id=923195 We are also experiencing this problem in 3.2.1 and 3.2.2. As a part of our solution we need to redeploy an ear file quite often and having to restart the server does not seem to be a nice fix. The memory leak occurs even if I remove all entries from application.xml. Nothing is being deployed, but the memory foot-print keeps growing... There seems to be two memory leaks: when runnig on Linux the top app reports SIZE and SHARE to grow with every redeploy. BTW, as a way to reproduce the problem I was redeploying web-console.war and that also resulted in momory usage growing, although at a much slower pace. ---------------------------------------------------------------------- Comment By: Jonathan Kovacs (hal200) Date: 2003-11-12 22:39 Message: Logged In: YES user_id=907916 I'm getting the same problem with JBoss 3.2.1_Tomcat-4.1.24. In my case, it seems to be triggered by a strange interaction with the Struts libraries. I threw together a quick script which continuously re-deploys the struts-example.war file over and over again (with a 15 second pause between deploy/undeployments) by popping the file in and out of the deploy directory (instead of using the deployer.jar tool...not sure at this point if that would make a difference) Sure enough, when watching the GC output, the memory use keeps increasing. If I stop the script, the memory doesn't seem to ever get reclaimed. If I leave it running, eventually the server blows it's heap and I start getting OutOfMemory errors at random intervals. ------------------------------------------------------------------------------- 17:05:45,223 ERROR [URLDeploymentScanner] Failed to deploy: org.jboss.deployment.scanner.URLDeploymentScanner$DeployedURL@32c13350{ url=file:/home/jon/cvs-test/canarie/lib/jboss-3.2.1_tomcat-4.1.24/server/canarie/deploy/IQXWeb.war, deployedLastModified=0 } org.jboss.deployment.DeploymentException: Could not create deployment: file:/home/jon/cvs-test/canarie/lib/jboss-3.2.1_tomcat-4.1.24/server/canarie/deploy/IQXWeb.war; - nested throwable: (java.lang.OutOfMemoryError) at org.jboss.deployment.MainDeployer.start(MainDeployer.java:853) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:640) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:613) at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177) at $Proxy7.deploy(Unknown Source) at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:302) at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:476) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:200) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:211) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:190) Caused by: java.lang.OutOfMemoryError ------------------------------------------------------------------------------- After leaving it for about an hour in this state, with GC details turned on, we can see that sure enough, the memory is not being reclaimed, despite the JVM's best efforts... Full GC [Tenured: 58303K->58303K(58304K), 0.3335490 secs] 64404K->64354K(64832K), 0.3336620 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3131040 secs] 64407K->64354K(64832K), 0.3131720 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3122840 secs] 64397K->64354K(64832K), 0.3123480 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3455040 secs] 64397K->64354K(64832K), 0.3455610 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3205630 secs] 64397K->64354K(64832K), 0.3206770 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3148190 secs] 64398K->64354K(64832K), 0.3148690 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3124290 secs] 64397K->64354K(64832K), 0.3125430 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3098440 secs] 64397K->64354K(64832K), 0.3099050 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3316150 secs] 64397K->64354K(64832K), 0.3317300 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3237010 secs] 64397K->64354K(64832K), 0.3238340 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3266160 secs] 64397K->64354K(64832K), 0.3266670 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.4335870 secs] 64397K->64354K(64832K), 0.4336530 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3097620 secs] 64397K->64354K(64832K), 0.3098310 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3138060 secs] 64398K->64354K(64832K), 0.3139190 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3125410 secs] 64397K->64354K(64832K), 0.3125970 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3188110 secs] 64397K->64354K(64832K), 0.3189260 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3175270 secs] 64397K->64354K(64832K), 0.3175970 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3120930 secs] 64397K->64354K(64832K), 0.3121500 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3116510 secs] 64397K->64354K(64832K), 0.3117170 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3098090 secs] 64397K->64354K(64832K), 0.3098610 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3256830 secs] 64398K->64354K(64832K), 0.3257470 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3076290 secs] 64397K->64354K(64832K), 0.3076920 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3166820 secs] 64397K->64354K(64832K), 0.3167470 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.5335920 secs] 64397K->64354K(64832K), 0.5336590 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3426230 secs] 64397K->64354K(64832K), 0.3427690 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.4453350 secs] 64397K->64354K(64832K), 0.4453950 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3081930 secs] 64397K->64354K(64832K), 0.3082590 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3183000 secs] 64398K->64354K(64832K), 0.3183600 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3121360 secs] 64397K->64354K(64832K), 0.3121970 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3134370 secs] 64397K->64354K(64832K), 0.3134990 secs] [Full GC [Tenured: 58303K->58303K(58304K), 0.3156870 secs] 64397K->64354K(64832K), 0.3157560 secs] ---------------------------------------------------------------------- Comment By: Heiko W.Rupp (pilhuhn) Date: 2003-09-29 09:17 Message: Logged In: YES user_id=217112 I also see the growth on 3.2.2rc5 as of this morning. With a 4MB ear I get the follwing memory consumption on Win2k with jdk 1.4.2_b28: -Xmx256m -XX:MaxPermSize=128m mem virt mem (xxx.000 MB as size unit) 92 116 105 127 118 144 130 152 142 167 150 174 then I stopped. No out of memory exception so far. -Xmx256m -XX:MaxPermSize=128m mem virt mem (xxx.000 MB as size unit) 89 94 100 112 114 130 123 143 131 152 139 160 then I stopped. Also no oom exception. ---------------------------------------------------------------------- Comment By: Juha Lindfors (juhalindfors) Date: 2003-09-26 17:35 Message: Logged In: YES user_id=175239 Does changing the GC permanent collection size have an effect on how quickly you get OutofMemoryException? JDK 1.4.2, -XX:MaxPermSize=<nn>m ---------------------------------------------------------------------- Comment By: Tim McCune (javajedi) Date: 2003-09-25 22:22 Message: Logged In: YES user_id=62441 This still happens as of version 3.2. ---------------------------------------------------------------------- Comment By: Juha Lindfors (juhalindfors) Date: 2003-09-25 21:51 Message: Logged In: YES user_id=175239 Resubmit with a more recent JBoss version if you believe this is still a problem. ---------------------------------------------------------------------- Comment By: Tim McCune (javajedi) Date: 2003-03-17 22:07 Message: Logged In: YES user_id=62441 This is still happening. See http://jboss.org/forums/thread.jsp?forum=121&thread=27888 Can you please reopen this bug? ---------------------------------------------------------------------- Comment By: Andreas Schaefer (schaefera) Date: 2001-10-30 02:07 Message: Logged In: YES user_id=70434 Need more information about what is deployed. Unfortunately memory is always limited and therefore hot deployment will one time reach a limit because all the loaded classes are not removed from the classloaders. But to see where a problem is I have to know what is deployed and maybe a error stack trace. ---------------------------------------------------------------------- Comment By: Andreas Schaefer (schaefera) Date: 2001-10-30 02:03 Message: Logged In: YES user_id=70434 Need more information about what is deployed. Unfortunately memory is always limited and therefore hot deployment will one time reach a limit because all the loaded classes are not removed from the classloaders. But to see where a problem is I have to know what is deployed and maybe a error stack trace. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=376685&aid=435958&group_id=22866 |