Memory consumption

[christophe.marchand]

Hi,

I use with success mysql-je, but my app seems to use a huge qty of memory : under Win32, the task manager/processes shows my java process memory use decreasing a lot when I iconify my app - which is normal, but the global view (all system) of memory use doesn't seem to decrease when I iconify.

Does the embedded engine allocate memory outside of java process ?

Does flushing table will deallocate memory used by engine ?

Thanks in advance for help,

Christophe

[Peter Schäfer]

> Does the embedded engine allocate memory outside of java process ?

yes.

Firstly, the embedded server allocates memory, but I have little insight into MySQL's memory management. I guess that the embedded server behaves quite similar to a normal MySQL server.

Secondly, the JDBC driver allocates Java heap memory _and_ native memory.
PreparedStatements should be closed when they are no longer needed. And, of course, unused Connections should be closed.

If you suspect there is a memory leak, please let me know how i could track it down.

> Does flushing table will deallocate memory used by engine ?

probably yes. However, flushing the tables has no effect on the JDBC layer (i.e. the code that I wrote).

[christophe.marchand]

I've checked my PreparedStatement and they are all correctly closed (that was a source of memory leaks).

My problem is the behavior is different between Win32 and Linux. With Linux, I've never seen memory consumption grow without limit. I have this behavior under Windows.

When I use a "normal" MySql server (ie not embedded), memory consumption stays very limited.

I think to many investigations :
- i do not give correct parameters to embedded server, so server consume all the available memory (and more...) ; I will have a look at parameters you use in Jose and mix them with "normal" my.ini server configuration
- i use a connection pool with 3 permanently openned connections to avoid stop and restart of embedded server. I will try to close connections while they are not used and mesure performance impact

If you want to test, a beta release is available at http://projets.admisource.gouv.fr/xemelios/ (windows-setup only), please contact my via private mail at christophe.marchand@axyus.com, I will send you data and a process to reproduce problem.

I will post a response to this thread with test results as soon as possible.

Christophe

[christophe.marchand]

Hi Peter,

Here are some tests I did :
Please consider the class Test :

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.Properties;

public class MySqlJETest {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("library.path","/home/chm/devel/XEMELIOS_ROOT/lib");
        props.put("--datadir","/home/chm/XemeliosMySql/MySql/data");
        props.put("--basedir","/home/chm/XemeliosMySql/MySql/mysql-programs");
        props.put("--default-character-set","utf8");
        props.put("--default-collation","utf8_general_ci");
        props.put("--skip-innodb","true");
        props.put("user","xemelios");
        props.put("password","xemelios");

        try {
            Class.forName("com.mysql.embedded.jdbc.MySqlEmbeddedDriver");
            Connection con = DriverManager.getConnection("jdbc:mysql-embedded/test",props);
            Statement st = con.createStatement();
            ResultSet rs = st.executeQuery("SHOW TABLES");
            while(rs.next()) System.out.println(rs.getString(1));
            rs.close();
            st.close();
            con.close();

            System.out.println();
           
            if(args.length>0) {
                int count = 0;
                try { count = Integer.parseInt(args[0]); } catch(Throwable t) {}
                for(int i=0;i<count;i++) {
                    Connection con2 = DriverManager.getConnection("jdbc:mysql-embedded/test",props);
                    st = con2.createStatement();
                    rs = st.executeQuery("SHOW TABLES");
                    while(rs.next()) System.out.println(rs.getString(1));
                    rs.close();
                    st.close();
                    con2.close();
                }
            }
        } catch(Throwable t) { t.printStackTrace(); }
    }

}

If you run test without arguments, everything's good. Then if you run "Test 1", two errors occurs :
first a java.lang.RuntimeException: out of memory is thrown.

Then, on a box without a nightrider user ;-)), this displays this message : 060504 16:11:55 [ERROR] Can't find messagefile '/home/nightrider/mysql50/share/mysql/english/errmsg.sys'   
--> seems there's something hard-coded somewhere in your code ;-))

Now, if you move the line "con.close();" just above the "catch" line, everything works better : it seems that once the server has been shutdown in a process it can not be restarted in this process. But while the server is still running, you can open and close as many connections as you want.

Last, in my application, i sometime get a jvm crash (with a core dump), but I can not manage to reproduce it in Test app.

Hope this will help to find buggs,

Christophe

[Peter Schäfer]

Thanks for your investigations !!

Your observation is correct: after closing the connection, the server process shuts down and I'm afraid it can't be launched a second time (at least not that easily...)

In the next release, I will include an option for a controlled shut down. In the meantime, just remember to keep one connection alive.

The hardcoded directory path results from the MySQL "configure" script, and I have not yet bothered to replace it ;-)
Usually, this error message indicates that the "--basedir" is missing.

[christophe.marchand]

I'm afraid --basedir is defined in this example