Menu

dbexport with java.lang.OutOfMemoryError: Java heap space

Help
Jerry
2018-03-19
2018-03-21
  • Jerry

    Jerry - 2018-03-19

    Hi,

    I have a big table that has over 80 millions records ,when I used the dbexport to export data ,I got an error about "java.lang.OutOfMemoryError: Java heap space" .
    I had adjusted the parameter of wrapper.java.initmemory and wrapper.java.maxmemory in sym_service.conf ,but the dbexport still got the out of memory error message.
    For this issue ,does it's correct to adjust these two parameters?
    Or which configuartion file/parameters should I adjust ?
    Thank you.

    The detail of error message:
    /mariadb/software/symmetric-server-3.9.4/bin/dbexport -e slave --compatible mysql --format CSV --no-create-info --dir /mariadb/data/symm/
    Log output will be written to /mariadb/software/symmetric-server-3.9.4/logs/symmetric.log
    [] - AbstractCommandLauncher - Option: name=engine, value={slave}
    [] - AbstractCommandLauncher - Option: name=compatible, value={mysql}
    [] - AbstractCommandLauncher - Option: name=format, value={CSV}
    [] - AbstractCommandLauncher - Option: name=no-create-info, value={}
    [] - AbstractCommandLauncher - Option: name=dir, value={/mariadb/data/symm/}
    [] - AbstractSymmetricEngine - Initializing connection to database
    [] - JdbcDatabasePlatformFactory - Detected database 'mariadb', version '5', protocol 'mysql'
    [] - JdbcDatabasePlatformFactory - The IDatabasePlatform being used is org.jumpmind.db.platform.mariadb.MariaDBDatabasePlatform
    java.lang.OutOfMemoryError: Java heap space
    Dumping heap to /mariadb/software/symmetric-server-3.9.4/tmp/java_pid3632.hprof ...
    Heap dump file created [574493609 bytes in 2.609 secs]
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3181)
    at java.util.ArrayList.grow(ArrayList.java:265)
    at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:239)
    at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:231)
    at java.util.ArrayList.add(ArrayList.java:462)
    at org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:207)
    at org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:195)
    at org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:185)
    at org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:121)
    at org.jumpmind.symmetric.io.data.DbExport.writeTable(DbExport.java:234)
    at org.jumpmind.symmetric.io.data.DbExport.exportTables(DbExport.java:189)
    at org.jumpmind.symmetric.io.data.DbExport.exportTables(DbExport.java:164)
    at org.jumpmind.symmetric.io.data.DbExport.exportTables(DbExport.java:138)
    at org.jumpmind.symmetric.DbExportCommand.executeWithOptions(DbExportCommand.java:190)
    at org.jumpmind.symmetric.AbstractCommandLauncher.execute(AbstractCommandLauncher.java:194)
    at org.jumpmind.symmetric.DbExportCommand.main(DbExportCommand.java:75)

     

    Last edit: Jerry 2018-03-19
  • Mark Michalek

    Mark Michalek - 2018-03-21

    Jerry, this sounds like a limitation of dbexport as it looks like all query results get loaded into memory around this line org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:207).

    You have a few options:
    1) Crank that wrapper.java.maxmemory as high as your system can handle (4g or 8g perhaps)
    2) Figure out a query that would allow you to segment the data a bit. That we you could do multiple extract passes and combin the file after the fact.
    3) Submit a pull request that handles the reading of rows & extracting in more of a streamed fashion, so that not so much memory is used. (https://github.com/JumpMind/symmetric-ds/compare?expand=1)
    4) Submit a bug/improvement report on the project's issue tracker: http://www.symmetricds.org/issues/view_all_bug_page.php

    Thanks,
    Mark

     

Log in to post a comment.