From: <mar...@pt...> - 2014-06-06 16:30:30
|
Hi, I am experiencing "memory problems" observing PVs over a long timespan. I've done the following: Started * custom CSS product based on 3.2.15 * NSLS-II 3.2.16a (downloaded from website) * SNS 3.2.15 (downloaded from website) Two pre-build products: * with a clean workspace * defaults as in the ini file * opened pv-table and added 20 "real" PVs (multiple current doubles) and 6 "sim" PVs Custom Build: * same epics config * custom table with same PVs connected updating table cells (at 4Hz each) OS: openSUSE 12.3 (x86_64) Kernel: 3.7.10-1.28-desktop VM: 1.7.0_51 OpenJDK 64-bit (later also tested with Oracle JDK, see below) The behaviour observed was that after nearly five days the memory (physical and swap) was almost full. Memory consumption according to "top" was around 30% each (Resident Memory > 2GB). Heap dumps of these processes showed "java.lang.ref.Finalizer" occupied > 50% (>16 MB) of the Heap. Finalizer statistics (Top Components->system class loader) revealed several objects of the following classes: * java.util.zip.ZipFile$ZipFileInflaterInputStream (thousands) * java.util.zip.ZipFile$ZipFileInputStream (thousands) * java.util.zip.ZipFile (hundreds) * java.util.zip.Inflater (hundreds) * java.net.SocksSocketImpl (varying hugely between applications) I have found some hints why this behavior might occur: Finalizers should generally be avoided (see Effective Java 2nd Edition, Bloch Item 7: "Avoid Finalizers"): Finalizer threads have low priority. If they cannot keep up with the rate of arriving objects (from threads with higher priority) queues grow -> heap fills. There's also a problem with allocated native memory which is not freed during finalization ? And if finalize fais/exits with errors it is not called again... The question is where does these zip objects come from. The Profiler showed several eclipse plugins (which are extracted from jars) as well as several SocksSocketImpl... Interesting Bug Report: http://bugs.java.com/view_bug.do?bug_id=4797189 ->CHANGING VM Due to these observations I decided to repeat the experiment but switching the used Java VM = Oracle JDK 1.7.0u60. Observations were a little different: "top" memory consumption: custom build: started with 3.0%, was 11.0% after 40h NSLS-II: started with 3.0%, was 3.8% after 40h SNS: started with 3.2%, was 13.5% after 40h I also observed that with this setup (Oracle VM) the heap size looked more like a sawtooth. When gc was active the consumption temporarily lowered a bit. Immediately after these events heap dumps temporarily looked better (less zip stuff). So I assume the effect is slower because gc on Oracle is more aggresive ? Another metric I observed was the "surviving generations". The 2 hours I observed them, they increased continually. When triggering GC manually they lowered a bit. A Netbeans Profiler article states that this metric should stabilize after a while: https://netbeans.org/kb/articles/nb-profiler-uncoveringleaks_pt1.html Maybe it has something to do with the native linux zip libraries used ? Does anyone have experienced similar behavior ? Or any hints ? Thanks Marcus |
From: <mar...@pt...> - 2014-06-05 16:10:19
|
Hi, I am experiencing "memory problems" observing PVs over a long timespan. I've done the following: Started * custom CSS product based on 3.2.15 * NSLS-II 3.2.16a (downloaded from website) * SNS 3.2.15 (downloaded from website) Two pre-build products: * with a clean workspace * defaults as in the ini file * opened pv-table and added 20 "real" PVs (multiple current doubles) and 6 "sim" PVs Custom Build: * same epics config * custom table with same PVs connected updating table cells (at 4Hz each) OS: openSUSE 12.3 (x86_64) Kernel: 3.7.10-1.28-desktop VM: 1.7.0_51 OpenJDK 64-bit (later also tested with Oracle JDK, see below) The behaviour observed was that after nearly five days the memory (physical and swap) was almost full. Memory consumption according to "top" was around 30% each (Resident Memory > 2GB). Heap dumps of these processes showed "java.lang.ref.Finalizer" occupied > 50% (>16 MB) of the Heap. Finalizer statistics (Top Components->system class loader) revealed several objects of the following classes: * java.util.zip.ZipFile$ZipFileInflaterInputStream (thousands) * java.util.zip.ZipFile$ZipFileInputStream (thousands) * java.util.zip.ZipFile (hundreds) * java.util.zip.Inflater (hundreds) * java.net.SocksSocketImpl (varying hugely between applications) I have found some hints why this behavior might occur: Finalizers should generally be avoided (see Effective Java 2nd Edition, Bloch Item 7: "Avoid Finalizers"): Finalizer threads have low priority. If they cannot keep up with the rate of arriving objects (from threads with higher priority) queues grow -> heap fills. There's also a problem with allocated native memory which is not freed during finalization ? And if finalize fais/exits with errors it is not called again... The question is where does these zip objects come from. The Profiler showed several eclipse plugins (which are extracted from jars) as well as several SocksSocketImpl... Interesting Bug Report: http://bugs.java.com/view_bug.do?bug_id=4797189 ->CHANGING VM Due to these observations I decided to repeat the experiment but switching the used Java VM = Oracle JDK 1.7.0u60. Observations were a little different: "top" memory consumption: custom build: started with 3.0%, was 11.0% after 40h NSLS-II: started with 3.0%, was 3.8% after 40h SNS: started with 3.2%, was 13.5% after 40h I also observed that with this setup (Oracle VM) the heap size looked more like a sawtooth. When gc was active the consumption temporarily lowered a bit. Immediately after these events heap dumps temporarily looked better (less zip stuff). So I assume the effect is slower because gc on Oracle is more aggresive ? Another metric I observed was the "surviving generations". The 2 hours I observed them, they increased continually. When triggering GC manually they lowered a bit. A Netbeans Profiler article states that this metric should stabilize after a while: https://netbeans.org/kb/articles/nb-profiler-uncoveringleaks_pt1.html Maybe it has something to do with the native linux zip libraries used ? Does anyone have experienced similar behavior ? Or any hints ? Thanks Marcus |
From: Kasemir, K. <kas...@or...> - 2014-06-06 18:19:41
|
Hi: In your description, I'm not sure which memory sizes refer to the overall memory usage as reported by ps, top, … Linux tools, and which are from jvisualvm, jprofiler, or even the plain CSS Monitor view, i.e. the JVM's idea of its memory usage. Do I understand correctly that with the Oracle JDK, the memory consumption is generally a sawtooth? Then I wouldn't worry about for example going from 3% to 11% after 40 hours, that could just mean: It settles in to 11%, that's what you see in "top", and within that overall memory size, the JVM goes up and down in a sawtooth manner. Completely running out of memory as you see with OpenJDK is of course a problem. If you can avoid that by using the Oracle JDK, then at least you have a workaround. You suggest that java.util.zip.* objects are using much of that memory. I don't see how CSS per se, tools like the PV Table would continually open ZIP files. ZIP files are used because Eclipse loads plugins, which are mostly *.jar, i.e. zip files. All that should happen when you start CSS, and whenever you for example open a PV table for the first time, so it loads the PV table plugin and dependencies. Still, I don't see that continuing all the time. When I check CSS running here, displaying *.opi files for Instruments, I see about 5000 java.util.zip.ZipFile$ZipFileInflaterInputStream instances, so it's indeed thousands as you found in your tests, but they are all removed after about a minute. The other case could be self-updates. When you go to Preferences, Install/Update, Automatic Updates, is that configured to periodically look for updates, which could mean that it connects to update sites, downloads the content.jar, which is a ZIP? -Kay On Jun 6, 2014, at 12:30 PM, mar...@pt... wrote: > Hi, > > I am experiencing "memory problems" observing PVs over a long timespan. I've done the following: > > Started > > * custom CSS product based on 3.2.15 > * NSLS-II 3.2.16a (downloaded from website) > * SNS 3.2.15 (downloaded from website) > > Two pre-build products: > * with a clean workspace > * defaults as in the ini file > * opened pv-table and added 20 "real" PVs (multiple current doubles) and 6 "sim" PVs > > Custom Build: > * same epics config > * custom table with same PVs connected updating table cells (at 4Hz each) > > OS: openSUSE 12.3 (x86_64) > Kernel: 3.7.10-1.28-desktop > VM: 1.7.0_51 OpenJDK 64-bit (later also tested with Oracle JDK, see below) > > The behaviour observed was that after nearly five days the memory (physical and swap) was almost full. > Memory consumption according to "top" was around 30% each (Resident Memory > 2GB). Heap dumps > of these processes showed "java.lang.ref.Finalizer" occupied > 50% (>16 MB) of the Heap. Finalizer > statistics (Top Components->system class loader) revealed several objects of the following classes: > > * java.util.zip.ZipFile$ZipFileInflaterInputStream (thousands) > * java.util.zip.ZipFile$ZipFileInputStream (thousands) > * java.util.zip.ZipFile (hundreds) > * java.util.zip.Inflater (hundreds) > * java.net.SocksSocketImpl (varying hugely between applications) > > I have found some hints why this behavior might occur: > > Finalizers should generally be avoided (see Effective Java 2nd Edition, Bloch Item 7: "Avoid Finalizers"): > Finalizer threads have low priority. If they cannot keep up with the rate of arriving objects (from threads > with higher priority) queues grow -> heap fills. > There's also a problem with allocated native memory which is not freed during finalization ? And if finalize > fais/exits with errors it is not called again... > > The question is where does these zip objects come from. The Profiler showed several eclipse plugins > (which are extracted from jars) as well as several SocksSocketImpl... > > Interesting Bug Report: > http://bugs.java.com/view_bug.do?bug_id=4797189 > > ->CHANGING VM > > Due to these observations I decided to repeat the experiment but switching the used Java VM = Oracle JDK 1.7.0u60. > Observations were a little different: > > "top" memory consumption: > custom build: started with 3.0%, was 11.0% after 40h > NSLS-II: started with 3.0%, was 3.8% after 40h > SNS: started with 3.2%, was 13.5% after 40h > > I also observed that with this setup (Oracle VM) the heap size looked more like a sawtooth. When gc was active the > consumption temporarily lowered a bit. Immediately after these events heap dumps temporarily looked better > (less zip stuff). So I assume the effect is slower because gc on Oracle is more aggresive ? > > Another metric I observed was the "surviving generations". The 2 hours I observed them, they increased continually. > When triggering GC manually they lowered a bit. A Netbeans Profiler article states that this metric should stabilize > after a while: > > https://netbeans.org/kb/articles/nb-profiler-uncoveringleaks_pt1.html > > Maybe it has something to do with the native linux zip libraries used ? > > Does anyone have experienced similar behavior ? Or any hints ? > > Thanks > Marcus------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/NeoTech_______________________________________________ > Cs-studio-users mailing list > Cs-...@li... > https://lists.sourceforge.net/lists/listinfo/cs-studio-users |
From: Carcassi, G. <car...@bn...> - 2014-06-09 13:08:27
|
>Does anyone have experienced similar behavior ? Or any hints ? Use jVisualVM (it's essentially the NB profiler bundled with the Oracle JDK). Start the memory profiler with the option to keep the allocation stack (that way we can know who generated it). Then create a memory dump, put it somewhere accessible so we can take a look. :-) Use of zip classes would normally be related to classloading. And the fact that if you change VM it behaves differently would not be surprising... but I'd like to see who allocates those objects before speculating... Gabriele |
From: <mar...@pt...> - 2014-06-13 16:56:56
|
Hi, the memory sizes were from the linux tool top and the heap dump MAT (Memory Analyzer Tools) of Eclipse as stated: - Memory consumption according to "top" was around... - Heap dumps of these processes showed... (MAT) - "top" memory consumption: Did I miss to declare it anywhere ? In the first test with OpenJDK I observed an increasing memory consumption (top) measured in percentage every ten minutes: https://github.com/eveCSS/profiling/blob/master/4356.png After four days MAT showed the zip stuff in the top components: https://github.com/eveCSS/profiling/blob/master/Finalizer-4356.png There were thousands of InputStreams (with closeRequested true) and referent=Finalizer: https://github.com/eveCSS/profiling/blob/master/Finalizer.png With the Oracle JDK the behavior was a bit better but not much (few less Finalizer objects...) I checked the hint with the self-updates. Since in our custom build version there is no update plugin this could not be the source. I repeated the experiment (although not as long as the first one which was ca. five days): With the OpenJDK I observed again a rising sawtooth of the heap memory: https://github.com/eveCSS/profiling/blob/master/19081-heap.png Again memory consumption (top) measured in percentage every ten minutes: https://github.com/eveCSS/profiling/blob/master/19081.png And with Oracle JDK: The heap lowered after a while starting again to rise: https://github.com/eveCSS/profiling/blob/master/13250-heap.png Memory Consumption according to top measured in percentage every ten minutes: https://github.com/eveCSS/profiling/blob/master/13250.png The Oracle JDK slightly "recovers" by freeing some memory after a long time, maybe the OpenJDK also will do it at a later time ? In the first run it did not... The second run was not as long as the first so maybe it was too early to observe, but I didn't see any "Live Objects" of the suspicious zip streams (except ca. 50 responsible for plugin jar loading). So 2nd time I got the same behavior as you. What was the difference to the first run ? Are there side effects which delay the Finalization of objects, esp. zip input streams, so long that they reside in memory (as mentioned in my first email -> Finalizers should generally be avoided) ? Maybe it is not connected to PVs itself but the amount of PVs updating themselves is (sometimes) so much "traffic" that the Finalizer thread does not run because of its low priority or because it is blocked ? The General Question is, do I have to worry ? ;-) Is there any hint of a memory leak or was it just bad circumstances ? Since I did not observe such many zip input streams I did not attach a memory dump (which also would be too large in many cases). Thanks Marcus Von: "Kasemir, Kay" <kas...@or...> An: "mar...@pt..." <mar...@pt...> Kopie: "<cs-...@li...>" <cs-...@li...> Datum: 06.06.2014 20:19 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs Hi: In your description, I'm not sure which memory sizes refer to the overall memory usage as reported by ps, top, ? Linux tools, and which are from jvisualvm, jprofiler, or even the plain CSS Monitor view, i.e. the JVM's idea of its memory usage. Do I understand correctly that with the Oracle JDK, the memory consumption is generally a sawtooth? Then I wouldn't worry about for example going from 3% to 11% after 40 hours, that could just mean: It settles in to 11%, that's what you see in "top", and within that overall memory size, the JVM goes up and down in a sawtooth manner. Completely running out of memory as you see with OpenJDK is of course a problem. If you can avoid that by using the Oracle JDK, then at least you have a workaround. You suggest that java.util.zip.* objects are using much of that memory. I don't see how CSS per se, tools like the PV Table would continually open ZIP files. ZIP files are used because Eclipse loads plugins, which are mostly *.jar, i.e. zip files. All that should happen when you start CSS, and whenever you for example open a PV table for the first time, so it loads the PV table plugin and dependencies. Still, I don't see that continuing all the time. When I check CSS running here, displaying *.opi files for Instruments, I see about 5000 java.util.zip.ZipFile$ZipFileInflaterInputStream instances, so it's indeed thousands as you found in your tests, but they are all removed after about a minute. The other case could be self-updates. When you go to Preferences, Install/Update, Automatic Updates, is that configured to periodically look for updates, which could mean that it connects to update sites, downloads the content.jar, which is a ZIP? -Kay On Jun 6, 2014, at 12:30 PM, mar...@pt... wrote: > Hi, > > I am experiencing "memory problems" observing PVs over a long timespan. I've done the following: > > Started > > * custom CSS product based on 3.2.15 > * NSLS-II 3.2.16a (downloaded from website) > * SNS 3.2.15 (downloaded from website) > > Two pre-build products: > * with a clean workspace > * defaults as in the ini file > * opened pv-table and added 20 "real" PVs (multiple current doubles) and 6 "sim" PVs > > Custom Build: > * same epics config > * custom table with same PVs connected updating table cells (at 4Hz each) > > OS: openSUSE 12.3 (x86_64) > Kernel: 3.7.10-1.28-desktop > VM: 1.7.0_51 OpenJDK 64-bit (later also tested with Oracle JDK, see below) > > The behaviour observed was that after nearly five days the memory (physical and swap) was almost full. > Memory consumption according to "top" was around 30% each (Resident Memory > 2GB). Heap dumps > of these processes showed "java.lang.ref.Finalizer" occupied > 50% (>16 MB) of the Heap. Finalizer > statistics (Top Components->system class loader) revealed several objects of the following classes: > > * java.util.zip.ZipFile$ZipFileInflaterInputStream (thousands) > * java.util.zip.ZipFile$ZipFileInputStream (thousands) > * java.util.zip.ZipFile (hundreds) > * java.util.zip.Inflater (hundreds) > * java.net.SocksSocketImpl (varying hugely between applications) > > I have found some hints why this behavior might occur: > > Finalizers should generally be avoided (see Effective Java 2nd Edition, Bloch Item 7: "Avoid Finalizers"): > Finalizer threads have low priority. If they cannot keep up with the rate of arriving objects (from threads > with higher priority) queues grow -> heap fills. > There's also a problem with allocated native memory which is not freed during finalization ? And if finalize > fais/exits with errors it is not called again... > > The question is where does these zip objects come from. The Profiler showed several eclipse plugins > (which are extracted from jars) as well as several SocksSocketImpl... > > Interesting Bug Report: > http://bugs.java.com/view_bug.do?bug_id=4797189 > > ->CHANGING VM > > Due to these observations I decided to repeat the experiment but switching the used Java VM = Oracle JDK 1.7.0u60. > Observations were a little different: > > "top" memory consumption: > custom build: started with 3.0%, was 11.0% after 40h > NSLS-II: started with 3.0%, was 3.8% after 40h > SNS: started with 3.2%, was 13.5% after 40h > > I also observed that with this setup (Oracle VM) the heap size looked more like a sawtooth. When gc was active the > consumption temporarily lowered a bit. Immediately after these events heap dumps temporarily looked better > (less zip stuff). So I assume the effect is slower because gc on Oracle is more aggresive ? > > Another metric I observed was the "surviving generations". The 2 hours I observed them, they increased continually. > When triggering GC manually they lowered a bit. A Netbeans Profiler article states that this metric should stabilize > after a while: > > https://netbeans.org/kb/articles/nb-profiler-uncoveringleaks_pt1.html > > Maybe it has something to do with the native linux zip libraries used ? > > Does anyone have experienced similar behavior ? Or any hints ? > > Thanks > Marcus------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/NeoTech_______________________________________________ > Cs-studio-users mailing list > Cs-...@li... > https://lists.sourceforge.net/lists/listinfo/cs-studio-users |
From: Carcassi, G. <car...@bn...> - 2014-06-13 18:20:17
|
Marcus: >The General Question is, do I have to worry ? ;-) In general, yes. :-) Now, in this _specific_ case... :-) >Is there any hint of a memory leak or was it just bad circumstances ? I don't see anything here that points to a leak. What you show seems to be consistent with a large max memory set compared to what is needed, and the GC never really bothers to do the deepest scan possible. Garbage collection is lazy, and in most cases the JVM will use all the memory you give it before doing the "deepest" GC scan. This is why, for example, is not recommended for UI type programs to allocate say 4 GB when 100MB is enough: it _will_ use the full 4 GB (even if it has no real use for it) and _then_ issue a deep GC that will freeze the whole UI to scan the whole memory... And with a heap that big it may take a couple of seconds. Things may have changed with Java 8, but the behavior is not uncommon. This trend: https://github.com/eveCSS/profiling/blob/master/13250-heap.png can be perfectly normal. The fastest sawtooth is a fast GC cycle limited to the first generation (objects that were just allocated). Then you see a slower sawtooth, with a deeper GC cycle. Then you have the even slower cycle (which you just one go), with the deepest GC cycle. The point is: just looking at the memory usage is not a good indicator. The most reliable indicator is comparing memory dumps: when that is done, unreachable objects are not included in the dump, so you know exactly what memory is actually used. Another way is to manually perform the GC: that's why you have a button there. You run the program for a while, perform GC, let it run again, perform GC again, and compare the two values after you performed GC. That way you are (more or less) comparing actual memory usage. I say more or less because there is no guarantee the GC will actually clean _all_. It may stop when it thinks it's "good enough". Note that for something like CS-Studio, where the memory usage changes drastically from run to run, this is a general problem with no solution: if you set the memory low, you may get a good memory footprint but the minute you open a large waveform (e.g. an image) you crash; if you set it high, you can use large waveforms, but if you don't you are going essentially to be wasting memory. This is even worse if you are trying to run in a shared environment (where each user will have his own instance). Gabriele From: mar...@pt... [mailto:mar...@pt...] Sent: Friday, June 13, 2014 12:57 PM To: Kasemir, Kay Cc: <cs-...@li...> Subject: Re: [Cs-studio-users] Filling Memory when observing PVs Hi, the memory sizes were from the linux tool top and the heap dump MAT (Memory Analyzer Tools) of Eclipse as stated: - Memory consumption according to "top" was around... - Heap dumps of these processes showed... (MAT) - "top" memory consumption: Did I miss to declare it anywhere ? In the first test with OpenJDK I observed an increasing memory consumption (top) measured in percentage every ten minutes: https://github.com/eveCSS/profiling/blob/master/4356.png After four days MAT showed the zip stuff in the top components: https://github.com/eveCSS/profiling/blob/master/Finalizer-4356.png There were thousands of InputStreams (with closeRequested true) and referent=Finalizer: https://github.com/eveCSS/profiling/blob/master/Finalizer.png<https://github.com/eveCSS/profiling/blob/master/Finalizer-4356.png> With the Oracle JDK the behavior was a bit better but not much (few less Finalizer objects...) I checked the hint with the self-updates. Since in our custom build version there is no update plugin this could not be the source. I repeated the experiment (although not as long as the first one which was ca. five days): With the OpenJDK I observed again a rising sawtooth of the heap memory: https://github.com/eveCSS/profiling/blob/master/19081-heap.png Again memory consumption (top) measured in percentage every ten minutes: https://github.com/eveCSS/profiling/blob/master/19081.png And with Oracle JDK: The heap lowered after a while starting again to rise: https://github.com/eveCSS/profiling/blob/master/13250-heap.png Memory Consumption according to top measured in percentage every ten minutes: https://github.com/eveCSS/profiling/blob/master/13250.png The Oracle JDK slightly "recovers" by freeing some memory after a long time, maybe the OpenJDK also will do it at a later time ? In the first run it did not... The second run was not as long as the first so maybe it was too early to observe, but I didn't see any "Live Objects" of the suspicious zip streams (except ca. 50 responsible for plugin jar loading). So 2nd time I got the same behavior as you. What was the difference to the first run ? Are there side effects which delay the Finalization of objects, esp. zip input streams, so long that they reside in memory (as mentioned in my first email -> Finalizers should generally be avoided) ? Maybe it is not connected to PVs itself but the amount of PVs updating themselves is (sometimes) so much "traffic" that the Finalizer thread does not run because of its low priority or because it is blocked ? The General Question is, do I have to worry ? ;-) Is there any hint of a memory leak or was it just bad circumstances ? Since I did not observe such many zip input streams I did not attach a memory dump (which also would be too large in many cases). Thanks Marcus Von: "Kasemir, Kay" <kas...@or...<mailto:kas...@or...>> An: "mar...@pt...<mailto:mar...@pt...>" <mar...@pt...<mailto:mar...@pt...>> Kopie: "<cs-...@li...<mailto:cs-...@li...>>" <cs-...@li...<mailto:cs-...@li...>> Datum: 06.06.2014 20:19 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs ________________________________ Hi: In your description, I'm not sure which memory sizes refer to the overall memory usage as reported by ps, top, ... Linux tools, and which are from jvisualvm, jprofiler, or even the plain CSS Monitor view, i.e. the JVM's idea of its memory usage. Do I understand correctly that with the Oracle JDK, the memory consumption is generally a sawtooth? Then I wouldn't worry about for example going from 3% to 11% after 40 hours, that could just mean: It settles in to 11%, that's what you see in "top", and within that overall memory size, the JVM goes up and down in a sawtooth manner. Completely running out of memory as you see with OpenJDK is of course a problem. If you can avoid that by using the Oracle JDK, then at least you have a workaround. You suggest that java.util.zip.* objects are using much of that memory. I don't see how CSS per se, tools like the PV Table would continually open ZIP files. ZIP files are used because Eclipse loads plugins, which are mostly *.jar, i.e. zip files. All that should happen when you start CSS, and whenever you for example open a PV table for the first time, so it loads the PV table plugin and dependencies. Still, I don't see that continuing all the time. When I check CSS running here, displaying *.opi files for Instruments, I see about 5000 java.util.zip.ZipFile$ZipFileInflaterInputStream instances, so it's indeed thousands as you found in your tests, but they are all removed after about a minute. The other case could be self-updates. When you go to Preferences, Install/Update, Automatic Updates, is that configured to periodically look for updates, which could mean that it connects to update sites, downloads the content.jar, which is a ZIP? -Kay On Jun 6, 2014, at 12:30 PM, mar...@pt...<mailto:mar...@pt...> wrote: > Hi, > > I am experiencing "memory problems" observing PVs over a long timespan. I've done the following: > > Started > > * custom CSS product based on 3.2.15 > * NSLS-II 3.2.16a (downloaded from website) > * SNS 3.2.15 (downloaded from website) > > Two pre-build products: > * with a clean workspace > * defaults as in the ini file > * opened pv-table and added 20 "real" PVs (multiple current doubles) and 6 "sim" PVs > > Custom Build: > * same epics config > * custom table with same PVs connected updating table cells (at 4Hz each) > > OS: openSUSE 12.3 (x86_64) > Kernel: 3.7.10-1.28-desktop > VM: 1.7.0_51 OpenJDK 64-bit (later also tested with Oracle JDK, see below) > > The behaviour observed was that after nearly five days the memory (physical and swap) was almost full. > Memory consumption according to "top" was around 30% each (Resident Memory > 2GB). Heap dumps > of these processes showed "java.lang.ref.Finalizer" occupied > 50% (>16 MB) of the Heap. Finalizer > statistics (Top Components->system class loader) revealed several objects of the following classes: > > * java.util.zip.ZipFile$ZipFileInflaterInputStream (thousands) > * java.util.zip.ZipFile$ZipFileInputStream (thousands) > * java.util.zip.ZipFile (hundreds) > * java.util.zip.Inflater (hundreds) > * java.net.SocksSocketImpl (varying hugely between applications) > > I have found some hints why this behavior might occur: > > Finalizers should generally be avoided (see Effective Java 2nd Edition, Bloch Item 7: "Avoid Finalizers"): > Finalizer threads have low priority. If they cannot keep up with the rate of arriving objects (from threads > with higher priority) queues grow -> heap fills. > There's also a problem with allocated native memory which is not freed during finalization ? And if finalize > fais/exits with errors it is not called again... > > The question is where does these zip objects come from. The Profiler showed several eclipse plugins > (which are extracted from jars) as well as several SocksSocketImpl... > > Interesting Bug Report: > http://bugs.java.com/view_bug.do?bug_id=4797189 > > ->CHANGING VM > > Due to these observations I decided to repeat the experiment but switching the used Java VM = Oracle JDK 1.7.0u60. > Observations were a little different: > > "top" memory consumption: > custom build: started with 3.0%, was 11.0% after 40h > NSLS-II: started with 3.0%, was 3.8% after 40h > SNS: started with 3.2%, was 13.5% after 40h > > I also observed that with this setup (Oracle VM) the heap size looked more like a sawtooth. When gc was active the > consumption temporarily lowered a bit. Immediately after these events heap dumps temporarily looked better > (less zip stuff). So I assume the effect is slower because gc on Oracle is more aggresive ? > > Another metric I observed was the "surviving generations". The 2 hours I observed them, they increased continually. > When triggering GC manually they lowered a bit. A Netbeans Profiler article states that this metric should stabilize > after a while: > > https://netbeans.org/kb/articles/nb-profiler-uncoveringleaks_pt1.html > > Maybe it has something to do with the native linux zip libraries used ? > > Does anyone have experienced similar behavior ? Or any hints ? > > Thanks > Marcus------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/NeoTech_______________________________________________ > Cs-studio-users mailing list > Cs-...@li...<mailto:Cs-...@li...> > https://lists.sourceforge.net/lists/listinfo/cs-studio-users |
From: Kasemir, K. <kas...@or...> - 2014-06-16 13:55:02
|
Hi: Thanks to Gabriele for the good description of general GC behavior. With either OpenJDK or the Oracle JDK, have you seen it crash with OutOfMemoryError? Or is it simply using memory towards the permitted limit? I am aware of a perm gen memory issue with Jython (https://github.com/ControlSystemStudio/cs-studio/issues/256, http://bugs.jython.org/issue1746). The CSS ticket is about the Scan Server, but it may be the same for Jython scripts inside BOY. The manifestation would be a pretty obvious java.lang.OutOfMemoryError: Permgen space. Thanks, Kay |
From: <mar...@pt...> - 2014-06-20 14:29:53
|
Hi, yes thanks too for the description although I feel a little more helpless now ;-) I didn't run the Applications long enough to tell if it crashes or not. Once I ran some in parallel (different versions and VMs) but since it was my normal working machine and it started to swap memory it wasn't usable in that state. So I had to stop. Another one (custom build v3.2.15, OpenJDK 7) is running for a week now and linux-top shows the following: KiB Mem: 8125900 total, 7843444 used, 282456 free, 18452 buffers KiB Swap: 2103292 total, 1623244 used, 480048 free, 768356 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21744 xyz 20 0 6211m 3,5g 11m S 8,6 45,6 1307:53 java Scenario is still the same (start the program, add ca. 20 PVs updating its values at 4Hz). Maybe this one will grab the "rest" of the free memory until Monday and we will see what happens then. Hopefully the linux oom-killer does not interfer. Marcus Von: "Kasemir, Kay" <kas...@or...> An: "mar...@pt..." <mar...@pt...> Kopie: "Carcassi, Gabriele" <car...@bn...>, "cs-...@li..." <cs-...@li...> Datum: 16.06.2014 15:55 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs Hi: Thanks to Gabriele for the good description of general GC behavior. With either OpenJDK or the Oracle JDK, have you seen it crash with OutOfMemoryError? Or is it simply using memory towards the permitted limit? I am aware of a perm gen memory issue with Jython ( https://github.com/ControlSystemStudio/cs-studio/issues/256, http://bugs.jython.org/issue1746). The CSS ticket is about the Scan Server, but it may be the same for Jython scripts inside BOY. The manifestation would be a pretty obvious java.lang.OutOfMemoryError: Permgen space. Thanks, Kay |
From: Kasemir, K. <kas...@or...> - 2014-06-20 14:46:55
|
Hi: This: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21744 xyz 20 0 6211m 3,5g 11m S 8,6 45,6 1307:53 java may just mean that you allowed too much memory, and it's using that. In comparison, I have CSS running on a control room computer for about 40 days: Cpu(s): 6.6%us, 2.8%sy, 0.0%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 3866144k total, 3637028k used, 229116k free, 366980k buffers Swap: 8339448k total, 88928k used, 8250520k free, 1745640k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5313 tgt-hall 20 0 3702m 500m 13m S 45.7 13.3 62287:11 /usr/java/current/bin/java -Xmx1024m -XX:MaxPermSize=128M -Dorg.apache.commons.logging.Log=org.apache.commons.logging.i Note how it was started with -Xmx1024m, -XX:MaxPermSize=128M. Memory usage is not a problem. In our case, on Linux,'%CPU is more of a concern, which is a separate issue that's in the process of being addressed. Thanks, Kay On Jun 20, 2014, at 10:29 AM, <mar...@pt...<mailto:mar...@pt...>> wrote: Hi, yes thanks too for the description although I feel a little more helpless now ;-) I didn't run the Applications long enough to tell if it crashes or not. Once I ran some in parallel (different versions and VMs) but since it was my normal working machine and it started to swap memory it wasn't usable in that state. So I had to stop. Another one (custom build v3.2.15, OpenJDK 7) is running for a week now and linux-top shows the following: KiB Mem: 8125900 total, 7843444 used, 282456 free, 18452 buffers KiB Swap: 2103292 total, 1623244 used, 480048 free, 768356 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21744 xyz 20 0 6211m 3,5g 11m S 8,6 45,6 1307:53 java Scenario is still the same (start the program, add ca. 20 PVs updating its values at 4Hz). Maybe this one will grab the "rest" of the free memory until Monday and we will see what happens then. Hopefully the linux oom-killer does not interfer. Marcus Von: "Kasemir, Kay" <kas...@or...<mailto:kas...@or...>> An: "mar...@pt...<mailto:mar...@pt...>" <mar...@pt...<mailto:mar...@pt...>> Kopie: "Carcassi, Gabriele" <car...@bn...<mailto:car...@bn...>>, "cs-...@li...<mailto:cs-...@li...>" <cs-...@li...<mailto:cs-...@li...>> Datum: 16.06.2014 15:55 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs ________________________________ Hi: Thanks to Gabriele for the good description of general GC behavior. With either OpenJDK or the Oracle JDK, have you seen it crash with OutOfMemoryError? Or is it simply using memory towards the permitted limit? I am aware of a perm gen memory issue with Jython (https://github.com/ControlSystemStudio/cs-studio/issues/256, http://bugs.jython.org/issue1746). The CSS ticket is about the Scan Server, but it may be the same for Jython scripts inside BOY. The manifestation would be a pretty obvious java.lang.OutOfMemoryError: Permgen space. Thanks, Kay |
From: <mar...@pt...> - 2014-06-25 17:40:05
|
Hi, I started the process with -Xmx256m but without any definition for perm size. Does this mean I am allowing (theoretically) infinite perm gen space for the process/vm ? But does this matter since in perm gen there are classes, Strings and other static stuff and the dynamically allocated space for objects etc. ist stored on the heap (which has a limit) ? I will try to repeat the tests with the perm gen size limit and see what happens... Curious what will happen when using Java 8 where the perm gen limit parameter has been removed: http://openjdk.java.net/jeps/122 . P.S. The 45,6% Memory process stopped running because the linux oom-killer process decided to stop it: [So Jun 22 21:09:41 2014] Out of memory: Kill process 21744 (java) score 474 or sacrifice child [So Jun 22 21:09:41 2014] Killed process 21744 (java) total-vm:7278584kB, anon-rss:4446556kB, file-rss:968kB Marcus Von: "Kasemir, Kay" <kas...@or...> An: "<mar...@pt...> " <mar...@pt...> Kopie: "Kasemir, Kay" <kas...@or...>, "Carcassi, Gabriele" <car...@bn...>, "cs-...@li..." <cs-...@li...> Datum: 20.06.2014 16:46 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs Hi: This: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21744 xyz 20 0 6211m 3,5g 11m S 8,6 45,6 1307:53 java may just mean that you allowed too much memory, and it's using that. In comparison, I have CSS running on a control room computer for about 40 days: Cpu(s): 6.6%us, 2.8%sy, 0.0%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 3866144k total, 3637028k used, 229116k free, 366980k buffers Swap: 8339448k total, 88928k used, 8250520k free, 1745640k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5313 tgt-hall 20 0 3702m 500m 13m S 45.7 13.3 62287:11 /usr/java/current/bin/java -Xmx1024m -XX:MaxPermSize=128M -Dorg.apache.commons.logging.Log=org.apache.commons.logging.i Note how it was started with -Xmx1024m, -XX:MaxPermSize=128M. Memory usage is not a problem. In our case, on Linux,'%CPU is more of a concern, which is a separate issue that's in the process of being addressed. Thanks, Kay On Jun 20, 2014, at 10:29 AM, <mar...@pt...< mailto:mar...@pt...>> wrote: Hi, yes thanks too for the description although I feel a little more helpless now ;-) I didn't run the Applications long enough to tell if it crashes or not. Once I ran some in parallel (different versions and VMs) but since it was my normal working machine and it started to swap memory it wasn't usable in that state. So I had to stop. Another one (custom build v3.2.15, OpenJDK 7) is running for a week now and linux-top shows the following: KiB Mem: 8125900 total, 7843444 used, 282456 free, 18452 buffers KiB Swap: 2103292 total, 1623244 used, 480048 free, 768356 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21744 xyz 20 0 6211m 3,5g 11m S 8,6 45,6 1307:53 java Scenario is still the same (start the program, add ca. 20 PVs updating its values at 4Hz). Maybe this one will grab the "rest" of the free memory until Monday and we will see what happens then. Hopefully the linux oom-killer does not interfer. Marcus Von: "Kasemir, Kay" <kas...@or...<mailto:kas...@or...>> An: "mar...@pt...<mailto:mar...@pt...>" <mar...@pt...<mailto:mar...@pt...>> Kopie: "Carcassi, Gabriele" <car...@bn...< mailto:car...@bn...>>, "cs-...@li...< mailto:cs-...@li...>" <cs-...@li...< mailto:cs-...@li...>> Datum: 16.06.2014 15:55 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs ________________________________ Hi: Thanks to Gabriele for the good description of general GC behavior. With either OpenJDK or the Oracle JDK, have you seen it crash with OutOfMemoryError? Or is it simply using memory towards the permitted limit? I am aware of a perm gen memory issue with Jython ( https://github.com/ControlSystemStudio/cs-studio/issues/256, http://bugs.jython.org/issue1746). The CSS ticket is about the Scan Server, but it may be the same for Jython scripts inside BOY. The manifestation would be a pretty obvious java.lang.OutOfMemoryError: Permgen space. Thanks, Kay |
From: Kasemir, K. <kas...@or...> - 2014-06-26 17:12:43
|
Hi: I don't understand. If you set -Xmx256m, that defines the overall memory limit. The JVM could stop because what you want to do is not possible with -Xmx256m. It shouldn't keep growing to the point where the Linux oom-killer becomes active. There may be some difference between what the JVM considers "256m" and what Linux sees for the overall process, but there's a huge difference between -Xmx256m = 256 MB and total-vm:7278584kB = 7GB. Maybe the -Xmx256m wasn't effective? Where did you specify it? As for perm gen space, there is a default limit built into the product. It's not unlimited. Perm gen space is as you say for classes, strings, ..,it doesn't tend to grow with runtime except for that Jython problem where each newly executed Jython file uses new perm gen space memory. Thanks, Kay On Jun 25, 2014, at 1:39 PM, mar...@pt... wrote: Hi, > > I started the process with -Xmx256m but without any definition for perm size. Does this mean I am allowing > (theoretically) infinite perm gen space for the process/vm ? But does this matter since in perm gen there > are classes, Strings and other static stuff and the dynamically allocated space for objects etc. ist stored on > the heap (which has a limit) ? > I will try to repeat the tests with the perm gen size limit and see what happens... > Curious what will happen when using Java 8 where the perm gen limit parameter has been removed: http://openjdk.java.net/jeps/122 . > > P.S. The 45,6% Memory process stopped running because the linux oom-killer process decided to stop it: > > [So Jun 22 21:09:41 2014] Out of memory: Kill process 21744 (java) score 474 or sacrifice child > [So Jun 22 21:09:41 2014] Killed process 21744 (java) total-vm:7278584kB, anon-rss:4446556kB, file-rss:968kB > > Marcus |
From: <mar...@pt...> - 2014-06-27 12:30:03
|
Hi, AFAIK -Xmx does not set the overall memory limit but the maximum heap size: >From http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html (Sizing the Generations) At initialization of the virtual machine, the entire space for the heap is reserved. The size of the space reserved can be specified with the -Xmx option. If the value of the -Xms parameter is smaller than the value of the -Xmx parameter, not all of the space that is reserved is immediately committed to the virtual machine. see also - http://stackoverflow.com/questions/6450132/java-seems-to-ignore-xms-and-xmx-options/6450260#6450260 - http://stackoverflow.com/questions/11801254/jvm-options-xms-and-xmx-are-ignored/11801546#11801546 The -Xmx256m was/is specified in the css.ini file: -vmargs -Xmx256m -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog -Dorg.apache.commons.logging.simplelog.defaultlog=fatal -Dno_proxy=127.0.0.1,localhost After starting it when using ps -ef | grep css I see the long outout with ... /usr/bin/java -vmargs -Xmx256m ... VisualVM also shows it in JVM arguments and in the monitor tab (as Gabriele showed in the screenshots): Heap: Max: 268.435.456 B Perm Gen Max: Max: 174.063.616 B For Perm Gen it seems to be really difficult to follow. What is also mentioned is - Code Cache (non-heap): The HotSpot Java VM also includes a code cache, containing memory that is used for compilation and storage of native code - Stack Size per thread: http://stackoverflow.com/questions/3513903/how-to-calculate-and-specify-the-total-memory-space-allowed-for-java-process/3515058#3515058 http://docs.oracle.com/javase/7/docs/technotes/guides/management/jconsole.html also says that non-heap "requires memory for internal processing or optimization". Marcus Von: "Kasemir, Kay" <kas...@or...> An: "mar...@pt..." <mar...@pt...> Kopie: "Kasemir, Kay" <kas...@or...>, "Carcassi, Gabriele" <car...@bn...>, "cs-...@li..." <cs-...@li...> Datum: 26.06.2014 19:12 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs Hi: I don't understand. If you set -Xmx256m, that defines the overall memory limit. The JVM could stop because what you want to do is not possible with -Xmx256m. It shouldn't keep growing to the point where the Linux oom-killer becomes active. There may be some difference between what the JVM considers "256m" and what Linux sees for the overall process, but there's a huge difference between -Xmx256m = 256 MB and total-vm:7278584kB = 7GB. Maybe the -Xmx256m wasn't effective? Where did you specify it? As for perm gen space, there is a default limit built into the product. It's not unlimited. Perm gen space is as you say for classes, strings, ..,it doesn't tend to grow with runtime except for that Jython problem where each newly executed Jython file uses new perm gen space memory. Thanks, Kay On Jun 25, 2014, at 1:39 PM, mar...@pt... wrote: Hi, > > I started the process with -Xmx256m but without any definition for perm size. Does this mean I am allowing > (theoretically) infinite perm gen space for the process/vm ? But does this matter since in perm gen there > are classes, Strings and other static stuff and the dynamically allocated space for objects etc. ist stored on > the heap (which has a limit) ? > I will try to repeat the tests with the perm gen size limit and see what happens... > Curious what will happen when using Java 8 where the perm gen limit parameter has been removed: http://openjdk.java.net/jeps/122 . > > P.S. The 45,6% Memory process stopped running because the linux oom-killer process decided to stop it: > > [So Jun 22 21:09:41 2014] Out of memory: Kill process 21744 (java) score 474 or sacrifice child > [So Jun 22 21:09:41 2014] Killed process 21744 (java) total-vm:7278584kB, anon-rss:4446556kB, file-rss:968kB > > Marcus |
From: Kasemir, K. <kas...@or...> - 2014-06-27 12:53:44
|
On Jun 27, 2014, at 8:29 AM, <mar...@pt...> wrote: > AFAIK -Xmx does not set the overall memory limit but the maximum heap size: … Right, there's some difference between the "256m" that the JVM provides to CSS for heap, and the overall memory footprint which includes the JVM itself and the perm gen space. Still, the latter two are a fairly fixed overhead, they shouldn't result in growing from "256m" to total-vm:7278584kB = 7GB. > The -Xmx256m was/is specified in the css.ini file: > ... > After starting it when using ps -ef | grep css I see the long outout with ... /usr/bin/java -vmargs -Xmx256m ... > VisualVM also shows it in JVM arguments and in the monitor tab (as Gabriele showed in the screenshots): > > Heap: Max: 268.435.456 B > Perm Gen Max: Max: 174.063.616 B Good, that confirms we have known Xmx settings. Overall, that makes it even more puzzling, and nobody else has reported such memory growth which seems to come from the JVM itself. If the heap or perm gen space ran into the configured limits, you should see out-of-memory exceptions from the JVM itself. So what's different about your setup? If you just run the /BOY Examples/1_2_WidgetExamples.opi that displays most widgets, using simulated PVs, does that grow over time for you? When talking to real EPICS PVs, are you using the pure Java CA client, or did you add the JNI version? Thanks, Kay |
From: <mar...@pt...> - 2014-06-30 14:29:32
|
Hi, that is what confused me too, an ever increasing memory consumption according to top but nothing in visualvm/jconsole/etc. The last test with the oom killer was custom build css with OpenJDK 1.7.0_u51... So quite a difference. The mystery is that not the java memory pools (heap, perm gen) are running out of memory but the vm process according to linux (ps/top and oom-killer). I am now running nsls2-3.2.16a with the boy widget examples... (custom build does not have boy yet) It is started with the integrated oracle vm 1.7.0_25. We will see. Epics was pure Java CA in all setups. Marcus Von: "Kasemir, Kay" <kas...@or...> An: "<mar...@pt...> " <mar...@pt...> Kopie: "cs-...@li..." <cs-...@li...> Datum: 27.06.2014 14:53 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs On Jun 27, 2014, at 8:29 AM, <mar...@pt...> wrote: > AFAIK -Xmx does not set the overall memory limit but the maximum heap size: ? Right, there's some difference between the "256m" that the JVM provides to CSS for heap, and the overall memory footprint which includes the JVM itself and the perm gen space. Still, the latter two are a fairly fixed overhead, they shouldn't result in growing from "256m" to total-vm:7278584kB = 7GB. > The -Xmx256m was/is specified in the css.ini file: > ... > After starting it when using ps -ef | grep css I see the long outout with ... /usr/bin/java -vmargs -Xmx256m ... > VisualVM also shows it in JVM arguments and in the monitor tab (as Gabriele showed in the screenshots): > > Heap: Max: 268.435.456 B > Perm Gen Max: Max: 174.063.616 B Good, that confirms we have known Xmx settings. Overall, that makes it even more puzzling, and nobody else has reported such memory growth which seems to come from the JVM itself. If the heap or perm gen space ran into the configured limits, you should see out-of-memory exceptions from the JVM itself. So what's different about your setup? If you just run the /BOY Examples/1_2_WidgetExamples.opi that displays most widgets, using simulated PVs, does that grow over time for you? When talking to real EPICS PVs, are you using the pure Java CA client, or did you add the JNI version? Thanks, Kay |
From: Carcassi, G. <car...@bn...> - 2014-06-30 14:57:57
|
Hi Marcus: In principle a leak in the JVM itself (e.g. JIT, GC, ...) would have that behavior. I am skeptical we'd bump into something like that ... But... Since Oracle/OpenJDK /6/7/8 are from the same codebase, you could try the IBM Java: http://www.ibm.com/developerworks/java/jdk/linux/download.html And see if you get any different behavior... You could also look at the full C dump... Never done it, though, so I'd have no idea even what to look for... :-/ Gabriele From: mar...@pt... [mailto:mar...@pt...] Sent: Monday, June 30, 2014 10:29 AM To: Kasemir, Kay Cc: cs-...@li... Subject: Re: [Cs-studio-users] Filling Memory when observing PVs Hi, that is what confused me too, an ever increasing memory consumption according to top but nothing in visualvm/jconsole/etc. The last test with the oom killer was custom build css with OpenJDK 1.7.0_u51... So quite a difference. The mystery is that not the java memory pools (heap, perm gen) are running out of memory but the vm process according to linux (ps/top and oom-killer). I am now running nsls2-3.2.16a with the boy widget examples... (custom build does not have boy yet) It is started with the integrated oracle vm 1.7.0_25. We will see. Epics was pure Java CA in all setups. Marcus Von: "Kasemir, Kay" <kas...@or...<mailto:kas...@or...>> An: "<mar...@pt...<mailto:mar...@pt...>> " <mar...@pt...<mailto:mar...@pt...>> Kopie: "cs-...@li...<mailto:cs-...@li...>" <cs-...@li...<mailto:cs-...@li...>> Datum: 27.06.2014 14:53 Betreff: Re: [Cs-studio-users] Filling Memory when observing PVs ________________________________ On Jun 27, 2014, at 8:29 AM, <mar...@pt...<mailto:mar...@pt...>> wrote: > AFAIK -Xmx does not set the overall memory limit but the maximum heap size: ... Right, there's some difference between the "256m" that the JVM provides to CSS for heap, and the overall memory footprint which includes the JVM itself and the perm gen space. Still, the latter two are a fairly fixed overhead, they shouldn't result in growing from "256m" to total-vm:7278584kB = 7GB. > The -Xmx256m was/is specified in the css.ini file: > ... > After starting it when using ps -ef | grep css I see the long outout with ... /usr/bin/java -vmargs -Xmx256m ... > VisualVM also shows it in JVM arguments and in the monitor tab (as Gabriele showed in the screenshots): > > Heap: Max: 268.435.456 B > Perm Gen Max: Max: 174.063.616 B Good, that confirms we have known Xmx settings. Overall, that makes it even more puzzling, and nobody else has reported such memory growth which seems to come from the JVM itself. If the heap or perm gen space ran into the configured limits, you should see out-of-memory exceptions from the JVM itself. So what's different about your setup? If you just run the /BOY Examples/1_2_WidgetExamples.opi that displays most widgets, using simulated PVs, does that grow over time for you? When talking to real EPICS PVs, are you using the pure Java CA client, or did you add the JNI version? Thanks, Kay |
From: Kasemir, K. <kas...@or...> - 2014-06-30 16:00:53
|
On Jun 30, 2014, at 10:57 AM, "Carcassi, Gabriele" <car...@bn...<mailto:car...@bn...>> wrote: Since Oracle/OpenJDK /6/7/8 are from the same codebase, you could try the IBM Java: http://www.ibm.com/developerworks/java/jdk/linux/download.html And see if you get any different behavior… We do have some code that uses *.sun.* classes which are in the Sun/Oracle runtimes, but may not be in others, see https://github.com/ControlSystemStudio/cs-studio/issues/576 So if you get class-not-found: *.sun.* type errors, that'd be the reason. Thanks, Kay |