Menu
▾
▴
Re: [dbunit-developer] Possible memory leak with DbUnitTask.java?
|
From: Roberto Lo G. <rlo...@gm...> - 2012-06-22 15:51:07
|
Or forking the execution, which starts a new separate JVM :-)
On Fri, Jun 22, 2012 at 2:01 PM, Shijun Kong <sk...@in...>wrote:
> As John and Roberto pointed out, PermGen is different from heap, a
> quick dirty way to fix the problem could be increase max PermGen size, -XX:MaxPermSize=512m.
> But per my experience, you won't get much after 512m.
>
> The root cause looks like in the way how ant execute task, seems like
> ant uses a new class loader for each call (<dbunit>) (see the exception
> trace). You should dig more in ant source code or move the looping into a
> shell script / junit test so as to manage the class loading by yourself.
>
> *
> __________________________
> *
> Shijun
>
> From: Roberto Lo Giacco <rlo...@gm...>
> Reply-To: "dbu...@li..." <
> dbu...@li...>
> To: "dbu...@li..." <
> dbu...@li...>
> Subject: Re: [dbunit-developer] Possible memory leak with DbUnitTask.java?
>
> John is right, the PermGen issue is not related to connection leak and
> if the connection variable is reused there is no leak: the heavyweight part
> of the connection is released within the close() method, the object itself,
> once closed, is lightweight and is marked for garbage collection once the
> variable is reused.
>
> The PermGen space is more probably related to the Ant task itself on
> which I'm not really expert as I do not use Ant since 2001... Can you
> create a pastebin with your Ant build script? I guess that if you are
> trying to run an external build without forking the JVM than your problem
> is related to Ant rather than dbUnit.
>
> Another check you can do is trying to execute the same amount of times
> in the same manner any other non trivial Ant task: if you have the same
> issue than you have the proof it's not a dbUnit problem.
>
> Cheers,
> Roberto
>
> On Fri, Jun 22, 2012 at 12:19 AM, John Hurst <joh...@gm...>wrote:
>
>> I thought that PermGen was used mostly for classes, rather than objects
>> such as Connections, which are stored on the normal heap. Actually I think
>> that traditionally -Xmx would not help with PermGen issues. But I vaguely
>> recall that some changes have occurred in the relationship between PermGen
>> and heap in recent years.
>>
>> Seems rather odd to me that leaking a IDatabaseConnection would really
>> cause this much harm. It's a rather trival resource.
>>
>> Most important thing is: Did your patch fix the issue for you? If so it
>> would tend to imply that the Connection leak was the problem...
>>
>> Regards
>>
>> John Hurst
>>
>> On Fri, Jun 22, 2012 at 10:04 AM, Charles Chan <cch...@ya...>wrote:
>>
>>> Hello,
>>>
>>> I was using dbunit's Ant integration and have a for loop that exports a
>>> bunch of dataset into many small xml files (less than 5-50kb each).
>>> After repeating <dbunit> for ~50 times, I got an OutOfMemoryException:
>>> PermGen space.
>>>
>>> (v2.4.8)
>>> C:\test\build.xml:364: java.lang.OutOfMemoryError: PermGen space
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>> at
>>> org.apache.tools.ant.AntClassLoader.defineClassFromData(AntClassLoader.java:1124)
>>> at
>>> org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.java:1295)
>>> at
>>> org.apache.tools.ant.AntClassLoader.findClassInComponents(AntClassLoader.java:1351)
>>> at
>>> org.apache.tools.ant.AntClassLoader.findClass(AntClassLoader.java:1311)
>>> at
>>> org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1064)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>> at
>>> org.apache.tools.ant.AntClassLoader.defineClassFromData(AntClassLoader.java:1124)
>>> at
>>> org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.java:1295)
>>> at
>>> org.apache.tools.ant.AntClassLoader.findClassInComponents(AntClassLoader.java:1351)
>>> at
>>> org.apache.tools.ant.AntClassLoader.findClass(AntClassLoader.java:1311)
>>> at
>>> org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1064)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>> at
>>> org.apache.tools.ant.AntClassLoader.defineClassFromData(AntClassLoader.java:1124)
>>> at
>>> org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.java:1295)
>>> at
>>> org.apache.tools.ant.AntClassLoader.findClassInComponents(AntClassLoader.java:1351)
>>> at
>>> org.apache.tools.ant.AntClassLoader.findClass(AntClassLoader.java:1311)
>>> at
>>> org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1064)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>> at
>>> org.apache.tools.ant.AntClassLoader.defineClassFromData(AntClassLoader.java:1124)
>>> at
>>> org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.java:1295)
>>>
>>> Although I can use -Xmx256m to increase the maximum heap space, I am
>>> curious why this problem occurred? I am able to import a 50MB xml file
>>> without specifying -Xmx option, so in theory, exporting 100 50kb xml
>>> files should NOT cause any problems. Also, the problem can be simulated
>>> by repeating the following ~100 times in a single Ant task.
>>>
>>> <target name="test-dbunit-export>
>>>
>>> <!-- export individual dataset -->
>>> <dbunit driver="${mysql.driver}"
>>> url="${mysql.url}/database"
>>> userid="${mysql.userid}"
>>> password="${mysql.password}"
>>> classpath="${mysql.classpath}">
>>> <export format="xml"
>>> dest="${dbunit.data.dir}/test1.xml">
>>> <query name="tableQuery" sql="SELECT * FROM table
>>> WHERE column=''"/>
>>> </export>
>>> </dbunit>
>>>
>>> <dbunit> ... </dbunit>
>>> <dbunit> ... </dbunit>
>>> <dbunit> ... </dbunit>
>>> ... <!-- repeat dbunit many times! -->
>>> <dbunit> ... </dbunit>
>>>
>>> </target>
>>>
>>> ----------
>>>
>>> Looking at the code, I noticed DbUnitTasks.java, in the method
>>> execute(), it
>>> only call "conn.close()", but what about "connection" object? Wouldn't
>>> that cause a leak if "connection" is holding a reference to "conn"?
>>> Should that be closed as well?
>>>
>>> I am not sure if I am on the right path, but have included a patch for
>>> review as well.
>>>
>>> Thank you!
>>> Charles
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> dbunit-developer mailing list
>>> dbu...@li...
>>> https://lists.sourceforge.net/lists/listinfo/dbunit-developer
>>>
>>>
>>
>>
>> --
>> Life is interfering with my game
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> dbunit-developer mailing list
>> dbu...@li...
>> https://lists.sourceforge.net/lists/listinfo/dbunit-developer
>>
>>
>
> ------------------------------
> This email message and any attachments may contain legally privileged or
> confidential information intended solely for the use of the individual or
> entity to whom it is addressed. If the reader of this message is not the
> intended recipient, you are hereby notified that any reading,
> dissemination, distribution or copying of this message or its attachments
> is strictly prohibited. If you have received this message in error, please
> notify us immediately by telephone, fax or email and delete the message and
> all attachments to the message. Any views or opinions expressed are solely
> those of the author and do not necessarily represent those of Investor
> Analytics LLC.
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> dbunit-developer mailing list
> dbu...@li...
> https://lists.sourceforge.net/lists/listinfo/dbunit-developer
>
>
|
