Menu
▾
▴
Re: [Wrapper-user] JVM hang - Xms / Xmx
|
From: Paul C. <cas...@au...> - 2004-03-31 23:44:19
|
Leif, Thanks for your info - I think I might have to buy a book and learn inside out about Java GC/memory usage. Our app takes around 950MB to start due to loading all the GIS stuff into RAM upon startup. The application author's theory was that reading the GIS stuff from RAM is heaps faster than from disk and it has to be read into RAM at some stage (when used), so it might as well stay there! Our app takes around 15 minutes to start on a fairly decent pair of boxes - the bottleneck is a gigabit Ethernet crossover cable running between the servers. Our client has ordered a pair of fibre gigabit cards for each box and a managed fibre switch, but they haven't turned up yet. The app takes 6 minutes to start when app and DB are on the same box, but that's less scalable (HD / RAM capacity constraints). Due to the startup time, we don't want to do anything to extend startup - that's why we're happy to allocate a fair chunk to the initial memory. It's interesting though - I've noticed that since reducing the initial memory value, the peak memory usage (according to Windows anyway) tends to remain pretty darn close to the initial memory value. This indicates that what you've outlined below seems to be working pretty well (ie when more RAM is needed, try a GC first). So what I've found is that by reducing the initial memory value in the config file, our JVM hasn't hung for the past 3 weeks. This is a good sign. The problem that's now occurring more frequently is the java.lang.OutOfMemoryError, which indicates that at some stage the process needs more RAM than is available: either the GC fails or something else is going on. And now that I think of it, we can't tell what the process maxes out at because by the time that I realise there's a problem the app has been restarted and Windows has lost the peak memory of the process that crashed (of course). I've turned on some logging as suggested to dump out memory usage, so we'll see what this shows. Incidently, the most I've been able to allocate to Java (Xmx) is about 1625MB on Windows 2000 Advanced Server. On NT (previous platform) it was only about 1100MB - although that was with JDK 1.3.0, so that may have had an impact. If I attempt starting with more than 1625MB, the app fails to start. I think this is just a Java/Windows relationship thing. The log files show the java.lang.OutOfMemoryError's for a while before the JVM eventually crashes. Is there anyway to set up the Wrapper so that I can be notified (preferably by email) of this error so that I can remote takeover the box and have a look at things before the Wrapper restarts it? Regards, Paul Casanova |---------+----------------------------------------> | | Leif Mortenson | | | <le...@ta...> | | | Sent by: | | | wra...@li...| | | ceforge.net | | | | | | | | | 31/03/2004 06:13 PM | | | Please respond to | | | wrapper-user | |---------+----------------------------------------> >--------------------------------------------------------------------------------------------------------------| | | | To: wra...@li... | | cc: | | Subject: Re: [Wrapper-user] JVM hang - Xms / Xmx | >--------------------------------------------------------------------------------------------------------------| Paul, Here is Java memory as I understand it. :-] As I understand it, the only reason to set the init memory value to anything other than the default is if you know that your application will always need a certain amount of memory on startup. This will cause the JVM to allocate all of that memory in one large chunk rather than bit by bit as the application loads and initializes all of its data. This can improve performance a bit. The JVM starts out with it allocated memory size set to the init memory size. If at any point, the JVM needs more memory than it has allocated then it first attempts to do a GC. If that fails then it will check to see if the allocated size is less than the max size. If it is then another block of memory is allocated, thus increasing the allocated memory. If the max size is hit then you will get an out of memory exception. This amount of real memory on the system does not cause OutOfMemory exceptions unless the max size set when launching the JVM is larger than the total Virtual memory on the machine. If you start using swap memory however then performance will be severely degraded (not an issue with you) Each time a GC is performed, old memory will be freed up from the allocated memory. As the JVM runs, its free memory will slowly decrease until the above is repeated. From what I have seen, once a block of memory is allocated from the system it is never returned. So the allocated memory will stick at the largest size since the JVM was started. (The exception appears to be 1.4.2 I have noticed that it seems to give memory back to the system?) Now, remember what I said about how the the JVM uses free memory until it is exhausted. Well if that free memory is very large then the time between GCs will be long. But the size of each GC will be quite large. This appears to cause a larger hiccup in JVM performance. For this reason, I usually try to keep the init memory small so that the free memory block is never any larger than is absolutely needed. This also makes sure that I don't use any more system memory than is really needed. So what happens if you don't set the initmemory at all. (Ie, leave it at 3MB) Your application should still work but it may not start up quite as fast. I am wondering how much memory the application takes on startup. Then what happens to the memory over time. It you have set the init memory to 1400 but it is still growing to 1600, this means that the application is really requiring that much memory. This could be because of a leak or because you actually need more than 1600MB. What happens if you set it to 2000MB Are you familiar with the memory methods in the Runtime class? Try setting up a thread in your program which logs the Allocated Memory, Free Memory, and then via a little math, the in use memory to the console once per minute or so. This will give you a much more accurate view of what is happening memory wise than you will get from watching the memory usage of Java in the Task Manager. You may also want to call System.gc just before taking the memory measurements to make sure you are looking at the memory usage without any garbage in memory. I always use the Instrumentation package that I wrote for the Apache Avalon Excalibur project to monitor memory and other data over time. It lets you view data like the images I attached. Much easier to understand than log output. That project is quite stable and well tested but at the moment it is lacking in documentation. Something I need to get around to doing. http://avalon.apache.org/excalibur/index.html More below. Paul Casanova wrote: >Thanks Leif. > >We have 6.3GB of RAM in a Win2K Advanced Server box - usually around 3GB >free, so there's no chance of running out of physical RAM. > >We were using: ># Initial Java Heap Size (in MB) >wrapper.java.initmemory=1600 > ># Maximum Java Heap Size (in MB) >wrapper.java.maxmemory=1600 > > >Now using: ># Initial Java Heap Size (in MB) >wrapper.java.initmemory=1400 > ># Maximum Java Heap Size (in MB) >wrapper.java.maxmemory=1600 > >When first trying the change, I tried initmemory=1200, then 1300, now 1400 >just to make sure it was functioning ok. In the Windows Task Manager the >Java process always peaked closed to each of these figures - never exceeded >1600MB as it did previously (used to get up to 1780MB: 1600MB + JVM >overhead I assume). > >I've seen this hotspot error once before since switching to JSW (months >back) - before hand it probably would have just been lost (svrany - MS >product) - yay for JSW I say. > >Once again I don't seem to be able to find anything concrete about this >error in Sun's Java forums, other than suggestions that native font dll's >may have become corrupt - but then one would expect the error to occur >regularly. It happened twice in one day last week, and then today, with >varying durations between the java.lang.OutOfMemoryError's and the JVM >crash. They've both been there each time though. > > I am wondering if 1600MB is too small or if your application is leaking memory? It would not be fun profiling an app that large. :-/ >We had always previously set the min and max heap values to the same. I >couldn't find much Sun info on how Xms and Xmx alter the behaviour of >garbage collection (or any other behaviour for that matter) - only in >http://java.sun.com/docs/hotspot/gc/. I just gave it a whirl and it seems >to have made a difference to the JVM hanging. > > What changes did you make? >I also read in one of the forums that issues with native calls from the JVM >were rectified in 1.4.1_02 +. Have you also heard this? > > I had not heard of any problems with earlier versions. Cheers, Leif #### IM.png has been removed from this note on April 01, 2004 by Paul Casanova |
