Re: [dbunit-developer] Possible memory leak with DbUnitTask.java?

[Roberto Lo G. <rlo...@gm...>]

John is right, the PermGen issue is not related to connection leak and if
the connection variable is reused there is no leak: the heavyweight part of
the connection is released within the close() method, the object itself,
once closed, is lightweight and is marked for garbage collection once the
variable is reused.

The PermGen space is more probably related to the Ant task itself on which
I'm not really expert as I do not use Ant since 2001... Can you create a
pastebin with your Ant build script? I guess that if you are trying to run
an external build without forking the JVM than your problem
is related to Ant rather than dbUnit.

Another check you can do is trying to execute the same amount of times in
the same manner any other non trivial Ant task: if you have the same issue
than you have the proof it's not a dbUnit problem.

Cheers,
   Roberto

On Fri, Jun 22, 2012 at 12:19 AM, John Hurst <joh...@gm...> wrote:

> I thought that PermGen was used mostly for classes, rather than objects
> such as Connections, which are stored on the normal heap. Actually I think
> that traditionally -Xmx would not help with PermGen issues. But I vaguely
> recall that some changes have occurred in the relationship between PermGen
> and heap in recent years.
>
> Seems rather odd to me that leaking a IDatabaseConnection would really
> cause this much harm. It's a rather trival resource.
>
> Most important thing is: Did your patch fix the issue for you? If so it
> would tend to imply that the Connection leak was the problem...
>
> Regards
>
> John Hurst
>
> On Fri, Jun 22, 2012 at 10:04 AM, Charles Chan <cch...@ya...> wrote:
>
>> Hello,
>>
>> I was using dbunit's Ant integration and have a for loop that exports a
>> bunch of dataset into many small xml files (less than 5-50kb each).
>> After repeating <dbunit> for ~50 times, I got an OutOfMemoryException:
>> PermGen space.
>>
>> (v2.4.8)
>> C:\test\build.xml:364: java.lang.OutOfMemoryError: PermGen space
>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>         at
>> org.apache.tools.ant.AntClassLoader.defineClassFromData(AntClassLoader.java:1124)
>>         at
>> org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.java:1295)
>>         at
>> org.apache.tools.ant.AntClassLoader.findClassInComponents(AntClassLoader.java:1351)
>>         at
>> org.apache.tools.ant.AntClassLoader.findClass(AntClassLoader.java:1311)
>>         at
>> org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1064)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>         at
>> org.apache.tools.ant.AntClassLoader.defineClassFromData(AntClassLoader.java:1124)
>>         at
>> org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.java:1295)
>>         at
>> org.apache.tools.ant.AntClassLoader.findClassInComponents(AntClassLoader.java:1351)
>>         at
>> org.apache.tools.ant.AntClassLoader.findClass(AntClassLoader.java:1311)
>>         at
>> org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1064)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>         at
>> org.apache.tools.ant.AntClassLoader.defineClassFromData(AntClassLoader.java:1124)
>>         at
>> org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.java:1295)
>>         at
>> org.apache.tools.ant.AntClassLoader.findClassInComponents(AntClassLoader.java:1351)
>>         at
>> org.apache.tools.ant.AntClassLoader.findClass(AntClassLoader.java:1311)
>>         at
>> org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1064)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>         at
>> org.apache.tools.ant.AntClassLoader.defineClassFromData(AntClassLoader.java:1124)
>>         at
>> org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.java:1295)
>>
>> Although I can use -Xmx256m to increase the maximum heap space, I am
>> curious why this problem occurred? I am able to import a 50MB xml file
>> without specifying -Xmx option, so in theory, exporting 100 50kb xml
>> files should NOT cause any problems. Also, the problem can be simulated
>> by repeating the following ~100 times in a single Ant task.
>>
>>         <target name="test-dbunit-export>
>>
>>                 <!-- export individual dataset -->
>>                 <dbunit driver="${mysql.driver}"
>>                         url="${mysql.url}/database"
>>                         userid="${mysql.userid}"
>>                         password="${mysql.password}"
>>                         classpath="${mysql.classpath}">
>>                    <export format="xml"
>> dest="${dbunit.data.dir}/test1.xml">
>>                       <query name="tableQuery" sql="SELECT * FROM table
>> WHERE column=''"/>
>>                    </export>
>>                  </dbunit>
>>
>>                  <dbunit> ... </dbunit>
>>                  <dbunit> ... </dbunit>
>>                  <dbunit> ... </dbunit>
>>                   ...  <!-- repeat dbunit many times! -->
>>                  <dbunit> ... </dbunit>
>>
>>       </target>
>>
>> ----------
>>
>> Looking at the code, I noticed DbUnitTasks.java, in the method execute(),
>> it
>> only call "conn.close()", but what about "connection" object? Wouldn't
>> that cause a leak if "connection" is holding a reference to "conn"?
>> Should that be closed as well?
>>
>> I am not sure if I am on the right path, but have included a patch for
>> review as well.
>>
>> Thank you!
>> Charles
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> dbunit-developer mailing list
>> dbu...@li...
>> https://lists.sourceforge.net/lists/listinfo/dbunit-developer
>>
>>
>
>
> --
> Life is interfering with my game
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> dbunit-developer mailing list
> dbu...@li...
> https://lists.sourceforge.net/lists/listinfo/dbunit-developer
>
>