#1262 STAX Service OutOfMemeory, even though -Xmx1g

Win32::Win2003
closed-fixed
Sharon Lucas
5
2010-07-30
2009-05-27
sfresher
No

I have a large XML file (104KB, 3000 lines) as my STAX execution workflow. I have constantly received OutOfMemory error on STAX service, even though I have specified "SERVICE STAX LIBRARY JSTAF EXECUTE {STAF/Config/STAFRoot}/services/stax/STAX.jar OPTION J2=-Xmx1g". The system has 4 GB of RAM.

Interestingly, using Windows Task Manager, I have found that the java.exe process always crash whenever it reaches about 150MB of memory consumption. It seems like the specified max heap size "1g" is not honored.

Discussion

  • Sharon Lucas
    Sharon Lucas
    2009-05-27

    • assigned_to: nobody --> slucas
     
  • Sharon Lucas
    Sharon Lucas
    2009-05-27

    What version of Java are you using to run the STAX service? Post the full version information provided in the JVMLog.1 file for the STAX service (found in {STAF/Config/STAFRoot}/data/STAF/lang/java/jvm/STAFJVM1/JVMLog.1).

    How much RAM is available?

    Note that its the JVM, not STAX, that should be honoring the maximum heap size you're specifying via the -Xmx1g option. Are sure the JVM you're using handles 1g for the maximum heap size? Did you try specifying -Xmx1024m instead?

     
  • Sharon Lucas
    Sharon Lucas
    2009-05-27

    Also, what version of STAF are you using on your STAX service machine?
    STAF local MISC VERSION

    And, what version of STAX are you using?
    STAF local STAX VERSION

    If you're using Sun Java 1.4.2, then see the the STAF FAQ at http://staf.sourceforge.net/current/STAFFAQ.htm. Question "3.1.1 When using Sun Java 1.4.2, why are the -Xmx settings for my Java STAF service not being used?" says:

    This appears to be a bug in Sun Java 1.4.2 where it is not using the -Xmx heap settings. This means that if you are running Java STAF services with Sun 1.4.2, and specifying the OPTION J2=-Xmx option, the option will not be used and your service JVM will run out of memory at a much lower heap size (the actual limit seems to vary by OS).

    IBM Java 1.4.2 works correctly (http://www-106.ibm.com/developerworks/java/jdk/index.html). Or try Sun Java 1.5 or 6.

     
  • Sharon Lucas
    Sharon Lucas
    2009-05-27

    Also, please provide your complete STAF.cfg file you're using on your STAX service machine.

     
  • sfresher
    sfresher
    2009-05-27

     
    Attachments
  • sfresher
    sfresher
    2009-05-27

    Thanks a lot for the the super fast response!

    - I have attached both STAX JVM log and STAF config file.
    - I am using the latest version of STAF and STAX.
    - I have tried both JRE5 and JRE6.
    - I have tried specifying "-Xmx1g", "-Xmx1500m", or "-Xmx1024m"
    - I have tried with or without "-Xrs"

     
  • sfresher
    sfresher
    2009-05-27

    Even when java.exe process is crashed, the system still have more than 3GB of free memory.

     
  • Sharon Lucas
    Sharon Lucas
    2009-05-27

    Very strange. We don't use the -Xrs option so you could try removing that.

    Also, to have your other Java services (FSExt, Event, SXE, Namespace) run in the AllOther JVM, you should specify OPTION JVMName=AllOther when registering these services in your STAF.cfg file.

    Note that we routinely run a much larger STAX XML file (672KB with 16,186 lines) with no problems on a Windows XP machine using the same Sun JVM and we specify OPTION J2=-Xmx1024m. Here's our JVMLog.1 info:

    ******************************************************************************
    *** 20090519-10:34:49 - Start of Log for JVMName: STAX
    *** JVM Executable: C:/sunjava50/bin/java
    *** JVM Options : -Xmx1024m
    *** JVM Version : java version "1.5.0_17"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_17-b04)
    Java HotSpot(TM) Client VM (build 1.5.0_17-b04, mixed mode)
    *** JVM PID : 4216
    ******************************************************************************

    Registered Extensions for STAX Version 3.3.6:

    Could you provide your STAX XML file do that we could try to recreate the problem?

     
  • sfresher
    sfresher
    2009-05-27

    I have attached the sample XML file. I was also wondering is this due to multi-thread processing? I have no process a few command, but when I send 150 concurrent commands that executes the sample XML, the JVM will just crash.

     
  • Sharon Lucas
    Sharon Lucas
    2009-05-27

    I can't run your STAX job since it depends on other files, etc being available.

    On your last comment, what do you mean by "when I send 150 concurrent commands that executes the sample XML"? Are you saying that in you run the attached STAX job, only executing this xml job once at a time, it runs fine. But, if you run the attached STAX job 150 times concurrently that the JVM crashes? If so, why are you running it 150 times concurrently? I could see where that could exceed the resources on your STAX machine.

     
  • sfresher
    sfresher
    2009-05-27

    Yes, I think you got what I mean.

    I developed the XML file as the workflow for testing our products in an automation process. Each developer can submit their own STAF commands, where they can specify the variables that is accepted in the XML file, such as the test script they want to run, the OS, the firmware, the hardware. I experienced that STAX cannot handle over 150 of these requests (STAX jobs) concurrently.

    I thought STAX can handle up to 10, 000 of jobs. Even though that could the theritical value, why the java.exe crashed at only 150MB, when I specify the max heap to be 1g?

     
  • Sharon Lucas
    Sharon Lucas
    2009-05-27

    Yes, STAX can handle many jobs, but you have to have enough system resources to support it. You may want to have more than one machine set up as a STAX service machine in order to handle 150 concurrent jobs.

    Note that it is the JVM, not STAX, that should be honoring the maximum heap
    size you're specifying via the -Xmx1024m option for the JVM when registering the STAX service. You can google for more information on Java OutOfMemory errors. You may want to try adjusting some other JVM options.

    Try setting the minimum heap size to the same value that you're setting the maximum heap size to. For example:

    OPTION -Xmx1536m OPTION -Xms1536m

    You may want to try selecting the "server" VM instead of the "client" VM (assuming you have the SDK, not just the JRE installed) by specifying:

    OPTION J2=-server

    Also, you could try reducing your stack size used for each thread by doing:

    OPTION J2=-Xss64k

    And, you may want to try some additional JVM options to provide for information on garbage collection and heap profiling. For example:

    OPTION J2=-verbose:gc
    OPTION J2=-Xrunhprof:heap=dump,format=a,file=c:/hprof-results.txt,doe=y

     
  • sfresher
    sfresher
    2009-05-27

    Thanks for all the details. I will keep on tuning it.

    However, it seems like the system does have enough resources to handle it: 3 out of 4 GB of RAM are free in all time.

    In addition, could you answer why "Even though that could
    the theritical value, why the java.exe crashed at only 150MB, when I
    specify the max heap to be 1g?"

     
  • Sharon Lucas
    Sharon Lucas
    2009-05-27

    I don't know why java.exe crashed at only 150MB when you specified a larger maximum heap size. That is a Java question, not a STAF/STAX question. You could ask it via a Java forum or google it...

     
  • sfresher
    sfresher
    2009-06-10

    Hi Sharon,

    Will I able to use other JVM (Oracle) other Sun's JVM? Will that deliver better performance? Possiblely?

     
  • Sharon Lucas
    Sharon Lucas
    2009-06-10

    The only JVMs that we have tested STAX with are IBM's and Sun's. You could try Oracle's and see.

     
  • Sharon Lucas
    Sharon Lucas
    2009-07-07

    I also suggest that you try the latest update for Sun Java 1.5.0 -- Update 19 (you were using Update 17), as this could be a problem in the JVM that's been fixed in the latest update package.

     
  • sfresher
    sfresher
    2009-07-07

    Hi Sharon,

    Thanks a lot! I had upgraded my JVM to JDK 6 update 14, so I don't need to switch back to JDK 5 for a better performance, do I?

    By the way, I have tuned my JVM with the following. This seems improved the performance a little bit.

    OPTION J2=-server OPTION J2=-Xms1024m OPTION J2=-Xmx1024m OPTION J2=-XX:PermSize=256m OPTION J2=-XX:MaxPermSize=256m OPTION J2=-Xss64k OPTION JVMName=STAX

     
  • Sharon Lucas
    Sharon Lucas
    2009-07-07

    No, you don't to switch back to JDK 5 since you're at the latest release for JDK 6. I'm glad you're seeing better performance now after tuning the JVM.

    So, are you no longer running out of memory? Can this bug be closed?

     
  • sfresher
    sfresher
    2009-07-07

    Since the JVM OutofMemory issue still exists, but less severe after tuning, I would suggest to downgrade the defect to a lower priority/severity, as we may continue tracking this issue.

     
  • Sharon Lucas
    Sharon Lucas
    2010-07-30

    • status: open --> closed-fixed
     
  • Sharon Lucas
    Sharon Lucas
    2010-07-30

    Closing since tuning Java for more memory resolved the issue.