the memory sizes were from the linux
tool top and the heap dump MAT (Memory Analyzer Tools) of Eclipse as stated:
- Memory consumption according to "top"
- Heap dumps of these processes showed... (MAT)
- "top" memory consumption:
Did I miss to declare it anywhere ?
In the first test with OpenJDK I observed an increasing
memory consumption (top) measured in percentage every ten minutes:
After four days MAT showed the zip stuff in the top
There were thousands of InputStreams (with closeRequested
true) and referent=Finalizer:
With the Oracle JDK the behavior was a bit better
but not much (few less Finalizer objects...)
I checked the hint with the self-updates. Since in
our custom build version there is no update plugin this could not be the
I repeated the experiment (although not as long as
the first one which was ca. five days):
With the OpenJDK I observed again a rising sawtooth
of the heap memory:
Again memory consumption (top) measured in percentage
every ten minutes:
And with Oracle JDK:
The heap lowered after a while starting again to rise:
Memory Consumption according to top measured in percentage
every ten minutes:
The Oracle JDK slightly "recovers" by freeing
some memory after a long time,
maybe the OpenJDK also will do it at a later time
? In the first run it did not...
The second run was not as long as the first so maybe
it was too early to observe, but I didn't see any "Live Objects"
the suspicious zip streams (except ca. 50 responsible
for plugin jar loading).
So 2nd time I got the same behavior as you. What was
the difference to the first run ?
Are there side effects which delay the Finalization
of objects, esp. zip input streams, so long
that they reside in memory (as mentioned in my first
email -> Finalizers should generally be avoided) ?
Maybe it is not connected to PVs itself but the amount
of PVs updating themselves is (sometimes)
so much "traffic" that the Finalizer thread
does not run because of its low priority or because it is blocked ?
The General Question is, do I have to worry ? ;-)
Is there any hint of a memory leak or was it just
bad circumstances ?
Since I did not observe such many zip input streams
I did not attach a memory dump (which also would be too large in many cases).
Filling Memory when observing PVs
In your description, I'm not sure which memory sizes refer to the overall
memory usage as reported by ps, top, … Linux tools, and which are from
jvisualvm, jprofiler, or even the plain CSS Monitor view, i.e. the JVM's
idea of its memory usage.
Do I understand correctly that with the Oracle JDK, the memory consumption
is generally a sawtooth?
Then I wouldn't worry about for example going from 3% to 11% after 40 hours,
that could just mean: It settles in to 11%, that's what you see in "top",
and within that overall memory size, the JVM goes up and down in a sawtooth
Completely running out of memory as you see with OpenJDK is of course
If you can avoid that by using the Oracle JDK, then at least you have a
You suggest that java.util.zip.* objects are using much of that memory.
I don't see how CSS per se, tools like the PV Table would continually open
ZIP files are used because Eclipse loads plugins, which are mostly *.jar,
i.e. zip files.
All that should happen when you start CSS, and whenever you for example
open a PV table for the first time, so it loads the PV table plugin and
dependencies. Still, I don't see that continuing all the time.
When I check CSS running here, displaying *.opi files for Instruments,
I see about 5000 java.util.zip.ZipFile$ZipFileInflaterInputStream instances,
so it's indeed thousands as you found in your tests, but they are all removed
after about a minute.
The other case could be self-updates.
When you go to Preferences, Install/Update, Automatic Updates, is that
configured to periodically look for updates, which could mean that it connects
to update sites, downloads the content.jar, which is a ZIP?
On Jun 6, 2014, at 12:30 PM, email@example.com wrote:
> I am experiencing "memory problems" observing PVs over a
long timespan. I've done the following:
> * custom CSS product based on 3.2.15
> * NSLS-II 3.2.16a (downloaded from website)
> * SNS 3.2.15 (downloaded from website)
> Two pre-build products:
> * with a clean workspace
> * defaults as in the ini file
> * opened pv-table and added 20 "real" PVs (multiple current
doubles) and 6 "sim" PVs
> Custom Build:
> * same epics config
> * custom table with same PVs connected updating table cells (at 4Hz
> OS: openSUSE 12.3 (x86_64)
> Kernel: 3.7.10-1.28-desktop
> VM: 1.7.0_51 OpenJDK 64-bit (later also tested with Oracle JDK, see
> The behaviour observed was that after nearly five days the memory
(physical and swap) was almost full.
> Memory consumption according to "top" was around 30% each
(Resident Memory > 2GB). Heap dumps
> of these processes showed "java.lang.ref.Finalizer" occupied
> 50% (>16 MB) of the Heap. Finalizer
> statistics (Top Components->system class loader) revealed several
objects of the following classes:
> * java.util.zip.ZipFile$ZipFileInflaterInputStream (thousands)
> * java.util.zip.ZipFile$ZipFileInputStream (thousands)
> * java.util.zip.ZipFile (hundreds)
> * java.util.zip.Inflater (hundreds)
> * java.net.SocksSocketImpl (varying hugely between applications)
> I have found some hints why this behavior might occur:
> Finalizers should generally be avoided (see Effective Java 2nd Edition,
Bloch Item 7: "Avoid Finalizers"):
> Finalizer threads have low priority. If they cannot keep up with the
rate of arriving objects (from threads
> with higher priority) queues grow -> heap fills.
> There's also a problem with allocated native memory which is not freed
during finalization ? And if finalize
> fais/exits with errors it is not called again...
> The question is where does these zip objects come from. The Profiler
showed several eclipse plugins
> (which are extracted from jars) as well as several SocksSocketImpl...
> Interesting Bug Report:
> ->CHANGING VM
> Due to these observations I decided to repeat the experiment but switching
the used Java VM = Oracle JDK 1.7.0u60.
> Observations were a little different:
> "top" memory consumption:
> custom build: started with 3.0%, was 11.0% after 40h
> NSLS-II: started with 3.0%, was 3.8% after 40h
> SNS: started with 3.2%, was 13.5% after 40h
> I also observed that with this setup (Oracle VM) the heap size looked
more like a sawtooth. When gc was active the
> consumption temporarily lowered a bit. Immediately after these events
heap dumps temporarily looked better
> (less zip stuff). So I assume the effect is slower because gc on Oracle
is more aggresive ?
> Another metric I observed was the "surviving generations".
The 2 hours I observed them, they increased continually.
> When triggering GC manually they lowered a bit. A Netbeans Profiler
article states that this metric should stabilize
> after a while:
> Maybe it has something to do with the native linux zip libraries used
> Does anyone have experienced similar behavior ? Or any hints
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases
> applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> Cs-studio-users mailing list