dbexport with java.lang.OutOfMemoryError: Java heap space

[Jerry]

Hi,

I have a big table that has over 80 millions records ,when I used the dbexport to export data ,I got an error about "java.lang.OutOfMemoryError: Java heap space" .
I had adjusted the parameter of wrapper.java.initmemory and wrapper.java.maxmemory in sym_service.conf ,but the dbexport still got the out of memory error message.
For this issue ,does it's correct to adjust these two parameters?
Or which configuartion file/parameters should I adjust ?
Thank you.

The detail of error message:
/mariadb/software/symmetric-server-3.9.4/bin/dbexport -e slave --compatible mysql --format CSV --no-create-info --dir /mariadb/data/symm/
Log output will be written to /mariadb/software/symmetric-server-3.9.4/logs/symmetric.log
[] - AbstractCommandLauncher - Option: name=engine, value={slave}
[] - AbstractCommandLauncher - Option: name=compatible, value={mysql}
[] - AbstractCommandLauncher - Option: name=format, value={CSV}
[] - AbstractCommandLauncher - Option: name=no-create-info, value={}
[] - AbstractCommandLauncher - Option: name=dir, value={/mariadb/data/symm/}
[] - AbstractSymmetricEngine - Initializing connection to database
[] - JdbcDatabasePlatformFactory - Detected database 'mariadb', version '5', protocol 'mysql'
[] - JdbcDatabasePlatformFactory - The IDatabasePlatform being used is org.jumpmind.db.platform.mariadb.MariaDBDatabasePlatform
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /mariadb/software/symmetric-server-3.9.4/tmp/java_pid3632.hprof ...
Heap dump file created [574493609 bytes in 2.609 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3181)
at java.util.ArrayList.grow(ArrayList.java:265)
at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:239)
at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:231)
at java.util.ArrayList.add(ArrayList.java:462)
at org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:207)
at org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:195)
at org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:185)
at org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:121)
at org.jumpmind.symmetric.io.data.DbExport.writeTable(DbExport.java:234)
at org.jumpmind.symmetric.io.data.DbExport.exportTables(DbExport.java:189)
at org.jumpmind.symmetric.io.data.DbExport.exportTables(DbExport.java:164)
at org.jumpmind.symmetric.io.data.DbExport.exportTables(DbExport.java:138)
at org.jumpmind.symmetric.DbExportCommand.executeWithOptions(DbExportCommand.java:190)
at org.jumpmind.symmetric.AbstractCommandLauncher.execute(AbstractCommandLauncher.java:194)
at org.jumpmind.symmetric.DbExportCommand.main(DbExportCommand.java:75)

[Mark Michalek]

Jerry, this sounds like a limitation of dbexport as it looks like all query results get loaded into memory around this line org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:207).

You have a few options:
1) Crank that wrapper.java.maxmemory as high as your system can handle (4g or 8g perhaps)
2) Figure out a query that would allow you to segment the data a bit. That we you could do multiple extract passes and combin the file after the fact.
3) Submit a pull request that handles the reading of rows & extracting in more of a streamed fashion, so that not so much memory is used. (https://github.com/JumpMind/symmetric-ds/compare?expand=1)
4) Submit a bug/improvement report on the project's issue tracker: http://www.symmetricds.org/issues/view_all_bug_page.php

Thanks,
Mark