character encoding issues & OOMs 1.8...

Help
KYTV I2P
2013-02-10
2014-01-19
  • KYTV I2P

    KYTV I2P - 2013-02-10

    Upgrading from 1.8.1 to 2.3.0 snapshot 20130206, and running into character encoding problems
    and strange OOMs.

    Not sure if there are encoding problems in our existing 1.8.1 database.

    Is there a change in UTF-8 handling/checking in 2.3.0?

    Is it possible that bad UTF-8 encoded LONGVARCHAR rows are causing a corrupt database after upgrade in case 1) below?

    Is it necessary to find and correct or purge the database of badly encoded LONGVARCHAR entries
    before or after upgrading to 2.3.0? How could we do that?

    Thanks for your help.


    Details:

    Upgrading from 1.8.1 to 2.2.9 works fine, but that has a problem with creating lobs
    due to this bug
    so we are trying the snapshot.

    We're doing SHUTDOWN COMPACT immediately after upgrading, as recommended.

    We have a table, call it aaa, that contans a LONGVARCHAR column.

    After upgrade, shutdown compact, and reconnecting, one of two things happens:


    case 1)

    OOM, even with -Xmx1024m on a 50 MB database, when hsqldb is processing the SET TABLE BBB INDEX line in the script file on reconnect.
    where this line is the line after the SET TABLE AAA INDEX line in the script file.

    java.sql.SQLException: error in script file line: 118 org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space in statement [SET TABLE BBB INDEX '240632 0 459']
        at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
        at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
        at org.hsqldb.jdbc.JDBCConnection.<init>(Unknown Source)
        at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
        at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:185)
        (... our code ...)
    Caused by: org.hsqldb.HsqlException: error in script file line: 118 org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space in statement [SET TABLE BBB INDEX '240632 0 459']
        at org.hsqldb.error.Error.error(Unknown Source)
        at org.hsqldb.scriptio.ScriptReaderText.readDDL(Unknown Source)
        at org.hsqldb.scriptio.ScriptReaderBase.readAll(Unknown Source)
        at org.hsqldb.persist.Log.processScript(Unknown Source)
        at org.hsqldb.persist.Log.open(Unknown Source)
        at org.hsqldb.persist.Logger.openPersistence(Unknown Source)
        at org.hsqldb.Database.reopen(Unknown Source)
        at org.hsqldb.Database.open(Unknown Source)
        at org.hsqldb.DatabaseManager.getDatabase(Unknown Source)
        at org.hsqldb.DatabaseManager.newSession(Unknown Source)
        ... 16 more
    Caused by: org.hsqldb.HsqlException: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
        at org.hsqldb.error.Error.error(Unknown Source)
        at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
        at org.hsqldb.persist.DataFileCache.get(Unknown Source)
        at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
        at org.hsqldb.persist.RowStoreAVLDisk.setAccessor(Unknown Source)
        at org.hsqldb.Table.setIndexRoots(Unknown Source)
        at org.hsqldb.Table.setIndexRoots(Unknown Source)
        at org.hsqldb.StatementCommand.getResult(Unknown Source)
        at org.hsqldb.StatementCommand.execute(Unknown Source)
        at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
        ... 25 more
    Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
        at org.hsqldb.error.Error.error(Unknown Source)
        at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)
        ... 34 more
    Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.hsqldb.rowio.RowInputBase.resetRow(Unknown Source)
        at org.hsqldb.rowio.RowInputBinary.resetRow(Unknown Source)
        at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)
        at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
        at org.hsqldb.persist.DataFileCache.get(Unknown Source)
        at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
        at org.hsqldb.persist.RowStoreAVLDisk.setAccessor(Unknown Source)
        at org.hsqldb.Table.setIndexRoots(Unknown Source)
        at org.hsqldb.Table.setIndexRoots(Unknown Source)
        at org.hsqldb.StatementCommand.getResult(Unknown Source)
        at org.hsqldb.StatementCommand.execute(Unknown Source)
        at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
        at org.hsqldb.scriptio.ScriptReaderText.readDDL(Unknown Source)
        at org.hsqldb.scriptio.ScriptReaderBase.readAll(Unknown Source)
        at org.hsqldb.persist.Log.processScript(Unknown Source)
        at org.hsqldb.persist.Log.open(Unknown Source)
        at org.hsqldb.persist.Logger.openPersistence(Unknown Source)
        at org.hsqldb.Database.reopen(Unknown Source)
        at org.hsqldb.Database.open(Unknown Source)
        at org.hsqldb.DatabaseManager.getDatabase(Unknown Source)
        at org.hsqldb.DatabaseManager.newSession(Unknown Source)
        at org.hsqldb.jdbc.JDBCConnection.<init>(Unknown Source)
        at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
        at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:185)
        (... our code ...)
    

    case 2)

    With a different version of the database, again, the upgrade, shutdown, and reconnect went fine.
    (This database version works fine when upgrading from 1.8.1 to 2.2.9).
    This time the startup went fine, but then:

    java.sql.SQLException: org.hsqldb.HsqlException: java.io.UTFDataFormatException in statement [ ALTER TABLE aaa ADD xyz TINYINT DEFAULT 0]
        at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
        at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
        at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
        at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source)
            (... our code ...)
    Caused by: org.hsqldb.HsqlException: org.hsqldb.HsqlException: java.io.UTFDataFormatException
        at org.hsqldb.error.Error.error(Unknown Source)
        at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
        at org.hsqldb.persist.DataFileCache.get(Unknown Source)
        at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
        at org.hsqldb.index.NodeAVLDisk.findNode(Unknown Source)
        at org.hsqldb.index.NodeAVLDisk.getLeft(Unknown Source)
        at org.hsqldb.index.IndexAVL.firstRow(Unknown Source)
        at org.hsqldb.persist.RowStoreAVL.rowIterator(Unknown Source)
        at org.hsqldb.persist.RowStoreAVL.moveData(Unknown Source)
        at org.hsqldb.TableWorks.moveData(Unknown Source)
        at org.hsqldb.TableWorks.addColumn(Unknown Source)
        at org.hsqldb.StatementSchema.getResult(Unknown Source)
        at org.hsqldb.StatementSchema.execute(Unknown Source)
        at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
        at org.hsqldb.Session.execute(Unknown Source)
        ... 16 more
    Caused by: org.hsqldb.HsqlException: java.io.UTFDataFormatException
        at org.hsqldb.error.Error.error(Unknown Source)
        at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
        ... 30 more
    Caused by: java.io.UTFDataFormatException
        at org.hsqldb.lib.StringConverter.readUTF(Unknown Source)
        at org.hsqldb.lib.StringConverter.readUTF(Unknown Source)
        at org.hsqldb.rowio.RowInputBinary.readString(Unknown Source)
        at org.hsqldb.rowio.RowInputBinary.readChar(Unknown Source)
        at org.hsqldb.rowio.RowInputBase.readData(Unknown Source)
        at org.hsqldb.rowio.RowInputBinary.readData(Unknown Source)
        at org.hsqldb.rowio.RowInputBase.readData(Unknown Source)
        at org.hsqldb.rowio.RowInputBinary.readData(Unknown Source)
        at org.hsqldb.rowio.RowInputBinaryDecode.readData(Unknown Source)
        at org.hsqldb.RowAVLDisk.<init>(Unknown Source)
        ... 31 more
    
     
  • Fred Toussi

    Fred Toussi - 2013-02-10

    If you send me the database, I will check and fix any issue. You can contact me via email (fredt at users.sourceforge.net).

     
  • KYTV I2P

    KYTV I2P - 2013-02-12

    Email sent. Thank you for your help with this. :)

     
  • KYTV I2P

    KYTV I2P - 2013-03-03

    Sorry for the delay in being able to check it out.

    It appears that the snapshot resolved the problem. Thanks for your help. :)

     
  • Fred Toussi

    Fred Toussi - 2013-03-03

    Thanks for reporting.

     

Log in to post a comment.