#120 Jacob memory leak

1.17
open
clay_shooter
memory (1)
5
2014-08-18
2014-03-19
Dima Gurov
No

Hi guys,
I have a memory leak in Jacob (not Java memory, but JVM process memory) when I'm parsing data returned by Excel (all that data I send from Excel to Java are accumulated in java.exe RAM - I've checked that by memory dump analysis).
I've tried jdk1.6.0_38, jdk1.7.0_51 (64 bit) with the same result.
Also I've followed recommendations from here: http://stackoverflow.com/questions/980483/jacob-doesnt-release-the-objects-properly
with no result.
Here is the way I use Jacob:

final Map<String, T> results = new HashMap<String, T>();
final VariantHolder sourceDataVariantHolder;
if (excelTable != null) {
    sourceDataVariantHolder = excelDispatcher.javaToExcelTable(excelTable);
} else {
    sourceDataVariantHolder = new VariantHolder(null, null, null);
}
final VariantHolder parametersVariantHolder = parameters.getParametersVariantHolder();
Variant tmp = null;
SafeArray safeArray = null;
try{
    ComThread.InitMTA();
    tmp = Dispatch.invoke(
                excel, 
                "Run", 
                Dispatch.Method, 
                new Object[] {
                        excelFunctionName,
                        parametersVariantHolder.getVariant(),
                        sourceDataVariantHolder.getVariant()
                        }, 
                new int[1]);
    safeArray = tmp.toSafeArray();
    final int rows = safeArray.getUBound(1) + 1;
    logger.info("Reading EXCEL String[][] array into a HashMap<String, String>");

    for (int row = 0; row < rows; row++) {
        final String key = safeArray.getString(row, 0);
        final T result = excelDispatcher.excelRowToJava(row, safeArray);
                ...
        results.put(key, result);
                ...
    }
}
finally{
    if(null != safeArray){
        safeArray.safeRelease();
    }
    if(null != tmp){
        tmp.safeRelease();
    }
    releaseVariantHolder(sourceDataVariantHolder);
    releaseVariantHolder(parametersVariantHolder);          

    // Added this to test Jacob clean-up.
    try {
        logger.info("Let's try to clean-up Jacob...");
        Class<ROT> rot = ROT.class;
        Method clear = rot.getDeclaredMethod("clearObjects", new Class[]{});
        clear.setAccessible(true);
        clear.invoke(null, new Object[]{});
        ComThread.Release();
        logger.info("Jacob clean-up complete");
    } catch( Exception ex ) {
        logger.error("Problem with Jacob clean-up.");
        ex.printStackTrace();
    }
}

return results;

Discussion

  • Dima, do you run this piece of code only once? Multiple runs should reuse the same memory. Does it work that way?

     
  • Dima Gurov
    Dima Gurov
    2014-03-19

    No, I run it repeatedly. Every time all data is accumulated in RAM - I saw 4 copies of my data in memory dump after 4 launches.

     
  • Please post a full application that exhibits the bug, so that we could try it in our environments. Without that it's hard to tell what's causing the memory growth. You may possibly accumulate the results in other part of your code.

    It would be best to use only WScripting objects or those that can be found on any Windows. But for me Excel is ok.

     
  • Dima Gurov
    Dima Gurov
    2014-03-20

    I see... Thanks for your reply.
    During my testing I cleaned up the results just after interaction with Jacob and still saw the leak.
    I will try to extract this part from application, but it's not very easy.

     
  • I probably got it. Run the code below and notice under task manager that every iteration of k consumes additional 36MB of memory. Is anyone able to fix this code so that it release the memory?

    import com.jacob.activeX.*;
    import com.jacob.com.*;
    
    public class MemLeak {
    
      public static void main(String[] args){
    
        for(int k=1; k<=5; k++) {
          System.out.println("start");
          ComThread.InitMTA();
          long a = 0;
          for (int j=1; j<20; j++)
          for (int i=1; i<=10000; i++) {
            Variant v = new Variant(3);
            a += v.getInt();
          }
          System.out.println("ready: " + a);
          try { Thread.sleep(5000); } catch (Throwable t){};
          System.out.println("releasing");
          ComThread.Release();
          try { Thread.sleep(5000); } catch (Throwable t){};
        }
      }
    }
    
     
  • That's weird. After removing Variant.finalize method memory usage reduces several times. And it's not a matter of safeRelease called there, because any finalize method, even as simple as cFin+=1, brings the memory leak back.

    Seems like a Java bug. Could we workaround this? By removing finalize method at all? ComThread.release calls safeRelease anyway. Dima, does this work for you?

     
  • ... or without jacob modification:

          for (int i=0; i<100; i++) {
            System.gc();
            System.runFinalization();
          }
    

    After running ComThread.release(). But this does not reduce the peak memory consumption, which is addressed by total removal of finalize method.

    Recommended reading: Jack Shirazi: The Secret Life Of The Finalizer

     
  • Dima Gurov
    Dima Gurov
    2014-03-24

    Thanks for research, I really appreciate it.

    I don't understand what do you mean by 'finalize method' - I do not call it, at least directly.

    Do you mean skipping the whole 'finally' section? Or just "safeArray.safeRelease()" and "tmp.safeRelease()" ?

     
    • I don't understand what do you mean by 'finalize method' - I do not call it, at least directly.

      You can modify Jacob source as suggested in this comment. If you want to use original Jacob, without modification, use this comment instead. The theory is described in the article, linked from the second comment.

       
  • Dima Gurov
    Dima Gurov
    2014-03-27

    Thanks, Jarek.
    Changing the Jacob itself is quite a big task for me, so I've decided to try the second option you've provided.
    I put 1000 cycles instead of 100 just to be on the safe side. Result is negative - the leak is still exist. I've attached memory consumption screenshot (25% CPU usage was for GC 1000 executions).
    NB I set -Xms2048m -Xmx2048m options in my Java, but Java.exe process go beyond that.

     
    Attachments
  • Dima Gurov
    Dima Gurov
    2014-03-27

    Also I want to notice that the leak is not in the heap memory, max heap size = 2,009,792 kbytes as you could see on the screenshot, but commited virtual memory is 2,576,276 kbytes (it could reach 12 GB and more, causing OOM error).

     
    Attachments
  • Dima Gurov
    Dima Gurov
    2014-03-28

    Jarek, your email doesn't working.

    I use jacob-1.17-M2-x64.dll

     
  • Dima, our private correspondence reveals that you use an old version of Jacob, 1.17-M2. You should always try using the newest version before submitting a bug report. If that's not possible, you should state which version you are using. Please test on 1.18-M1. Is it better?

     
  • And here is jacob.jar 1.18-M1 stripped from any finalizers. Just for the tests, not for production environment.

     
    Last edit: Jarek Czekalski 2014-03-31
    Attachments
  • ... and this is the diff file for this jar (as GNU license requires when publishing modified version).

     
    Last edit: Jarek Czekalski 2014-03-31
  • Dima Gurov
    Dima Gurov
    2014-04-01

    Thank you Jarek!
    I've tested 1.18 original version, and your fixed version, and the both had a leak. 130 MB and 126 MB respectively (average for 2 runs each).
    Do you have leak on your example?

     
    • Thanks for doing the testing, Dima. Now that I kind of fixed the problem in my example and this trick doesn't work for you - the only possibility to move on is that you post a complete code to reproduce the issue.

       
  • david crosson
    david crosson
    2014-05-07

    Hello, I encounter a memory leak (probably the same as the one described here) in my project "wmirp" : https://github.com/dacr/wmirp
    I've many Finalizer that are still pending,

    You can easily reproduce the memory leak by just starting (once compiled) wmirp as follow :
    java -jar wmirp.jar
    or
    java -jar target\scala-2.10\wmirp.jar

    I've generated a heap dump that I can make available to you.

     
    Attachments
  • Scott Johnson
    Scott Johnson
    2014-06-02

    Whether we use -Dcom.jacob.autogc or call .safeRelease(), the memory usage in Java just accumulates in java.exe and doesn't come down.

    Calling ComThread.Release() would release the memory but we have to traverse the data in one go.

    We're using Jacob 1.17 with jacob-1.17-x86.dll on 64-bit Windows 2008. Java 1.6 and 1.7 doesn't seem to make a difference.