[Cobalt-developers] Boss student and staff server memory leak search

[Harri <har...@tu...>]

Hello,

For the past few weeks we've been struggling with OutOfMemory issues.=20
The student and the staff server will keep on allocating more and more=20
memory during use. This is especially noticeable when there are lots of=20
active users.

In the past we had managed by increasing the maximum heap size of the=20
JVM and restarting the servers once in a while. But as the load on the=20
servers has been increasing it has been problematic. The memory will run=20
out several times a week.

I tried writing a unix script to kill the JVM when OutOfMemoryError
is written out to the standard output of the server, but the server has
usually performed very badly for 5 to several tens of minutes before=20
the OutOfMemory message is written out.=20

Catching the OutOfMemoryError is not possibly as it will always happen=20
in a RMI-thread where you cannot set up your own exception handlers.

Then I tried studying if there is a problem with the garbage collection.
I tried the parallel garbage collection algorithm and logging the=20
verbose garbage collection stats to a log. It was evident from the=20
output that even full garbage collections didn't reduce the heap size=20
significantly. It was clear that memory was leaked or as is the case in=20
the Java world there was a severe reference retention somewhere.

Next I wanted to look at the heap dump. What objects take all the heap=20
space. Running HAT and jmap on Sun's JVM 1.5 showed the following:

$ jmap -histo $PID | head

Size    Count   Class description
-------------------------------------------------------
3013136 70866   char[]
2132744 17288   * ConstMethodKlass
2012336 21929   byte[]
1597872 66578   java.lang.String
1501184 11728   com.mysql.jdbc.Field
1165584 48566   java.util.HashMap$Entry

HAT let me browse where those Field-objects were referenced. They were=20
stored in com.mysql.jdbc.Connection.

I thought it could be a problem in the MySQL Connector/J
(MySQL's JDBC Driver) version 3.1.7 that I was using. I downloaded=20
version 5.0.4. While reading the CHANGES to see what was new, I=20
searched for the word memory and found out the following bit in 3.1.7=20
changes:

   - Added 'dontTrackOpenResources' option (default is false, to be
      JDBC compliant), which helps with memory use for non-well-behaved
      apps (i.e applications which don't close Statements when they
      should).

I already had version 3.1.7 installed so I just tried to apply that
option. The corresponding line in data/settings/cobalt.properties

database_url=3Djdbc:mysql://localhost/boss?user=3Dboss&password=3Dsecret&=
dontTrackOpenResources=3Dtrue

After restarting the servers the heap size has stayed below 90 MB for=20
both servers.

This is only a workaround. To be JDBC compliant, the database objects=20
used by Boss should be closed.

The problem was about using the database, so it is understandable that=20
this doesn't affect the slave server which doesn't access the database.

Yours,
Harri J=E4rvi