#91 Excessive garbagecollection&negative reserved mem

[aregr01]

It seems that write access to the global counters for native memory usage is not properly synchronized. This is a problem since native memory deallocation happens in a separate thread (DisposableGC thread) from where the allocation happens (application thread). Also this would possibly lead to problems if one had several application threads doing allocation.

In practice, this missing synchronization seems to leads to negative values of the memory_usage.reserved_direct as reported from java. This has been reported repeatedly with nio.vverbose turned on, typically after allocating + deallocating 1-10 millions of packets.

On the C-native side, the memory counters are defined as a uint_64. Thus decrementing these counters below 0 basically creates a huge number.

Whenever this occurs, any consecutive call to allocate0 will observe a huge value of memory_usage.reserved_direct and thus cause a soft memory limit breach to be falsely detected. This in turn leads to an explicit call to System.gc(). Sadly,since the reserved_memory counter is already below 0, a GC basically increases the chance that the next call to also allocate also causes a soft breach, since a GC would spawn more deallocations.

In an actual application I have seen that System.gc calls get more and more frequent, finally stabilizing with one System.gc call each 200 ms (configured as maximum frequency by default)

This amount of excessive and unnecessary calls to System.gc gives a significant decrease in application performance and huge increase of CPU usage.
If a sniffing application, this typically leads to packet losses in pcap.

Steps to reproduce:

*Write an application that creates a lot of JMemory objects of size 1k and quickly discards them (so they are picked up by the DisposableGC thread)
*Run program with nio.vvverbose turned on
*Watch for output indicating soft limit breached and negative reserved_memory counters

To fix:
I have created a quick fix that adds synchronization around JMemory.allocate0 and JMemoyReference.disposeNative0.This seems to fix the issue

[aregr01]

Memeory allocation/deallocation synch fix

[Sly Technologies]

Thanks for the patch and detailed analysis. I will incorporate/improve memory allocation algorithm.

[Sly Technologies]

I looked at your patch and I had already done the synchornization around the allocate and dispose methods for another related bug. The only difference between your patch and my code is that I used synchronized on the static allocate0 method and a number of other static methods that also look up the memory allocation counters. The synchronized modifier on a static method has the effect of synchornizing on the JMemory class object and lower code overhead as there is huge amount of code generated by the compiler when you use synchronized blocks. In the dispose side we have to synchronize around JMemory.class in a block, no choice there.

Also I believe this bug is also responsible for other issues that have been reported in DisposableGC thread, such as invalid queue sizes as they are also dependent on those native memory counters JMemory class keeps.

I have everything checked into SVN (r1333) on the 1.3.1 branch (not the 1.3 branch.)