Upgrading from 1.8.1 to 2.3.0 snapshot 20130206, and running into character encoding problems
and strange OOMs.
Not sure if there are encoding problems in our existing 1.8.1 database.
Is there a change in UTF-8 handling/checking in 2.3.0?
Is it possible that bad UTF-8 encoded LONGVARCHAR rows are causing a corrupt database after upgrade in case 1) below?
Is it necessary to find and correct or purge the database of badly encoded LONGVARCHAR entries
before or after upgrading to 2.3.0? How could we do that?
Thanks for your help.
Details:
Upgrading from 1.8.1 to 2.2.9 works fine, but that has a problem with creating lobs
due to this bug
so we are trying the snapshot.
We're doing SHUTDOWN COMPACT immediately after upgrading, as recommended.
We have a table, call it aaa, that contans a LONGVARCHAR column.
After upgrade, shutdown compact, and reconnecting, one of two things happens:
case 1)
OOM, even with -Xmx1024m on a 50 MB database, when hsqldb is processing the SET TABLE BBB INDEX line in the script file on reconnect.
where this line is the line after the SET TABLE AAA INDEX line in the script file.
java.sql.SQLException: error in script file line: 118 org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space in statement [SET TABLE BBB INDEX '240632 0 459']
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCConnection.<init>(Unknown Source)
at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
(... our code ...)
Caused by: org.hsqldb.HsqlException: error in script file line: 118 org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space in statement [SET TABLE BBB INDEX '240632 0 459']
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.scriptio.ScriptReaderText.readDDL(Unknown Source)
at org.hsqldb.scriptio.ScriptReaderBase.readAll(Unknown Source)
at org.hsqldb.persist.Log.processScript(Unknown Source)
at org.hsqldb.persist.Log.open(Unknown Source)
at org.hsqldb.persist.Logger.openPersistence(Unknown Source)
at org.hsqldb.Database.reopen(Unknown Source)
at org.hsqldb.Database.open(Unknown Source)
at org.hsqldb.DatabaseManager.getDatabase(Unknown Source)
at org.hsqldb.DatabaseManager.newSession(Unknown Source)
... 16 more
Caused by: org.hsqldb.HsqlException: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
at org.hsqldb.persist.DataFileCache.get(Unknown Source)
at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
at org.hsqldb.persist.RowStoreAVLDisk.setAccessor(Unknown Source)
at org.hsqldb.Table.setIndexRoots(Unknown Source)
at org.hsqldb.Table.setIndexRoots(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
... 25 more
Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)
... 34 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.hsqldb.rowio.RowInputBase.resetRow(Unknown Source)
at org.hsqldb.rowio.RowInputBinary.resetRow(Unknown Source)
at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)
at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
at org.hsqldb.persist.DataFileCache.get(Unknown Source)
at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
at org.hsqldb.persist.RowStoreAVLDisk.setAccessor(Unknown Source)
at org.hsqldb.Table.setIndexRoots(Unknown Source)
at org.hsqldb.Table.setIndexRoots(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.scriptio.ScriptReaderText.readDDL(Unknown Source)
at org.hsqldb.scriptio.ScriptReaderBase.readAll(Unknown Source)
at org.hsqldb.persist.Log.processScript(Unknown Source)
at org.hsqldb.persist.Log.open(Unknown Source)
at org.hsqldb.persist.Logger.openPersistence(Unknown Source)
at org.hsqldb.Database.reopen(Unknown Source)
at org.hsqldb.Database.open(Unknown Source)
at org.hsqldb.DatabaseManager.getDatabase(Unknown Source)
at org.hsqldb.DatabaseManager.newSession(Unknown Source)
at org.hsqldb.jdbc.JDBCConnection.<init>(Unknown Source)
at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
(... our code ...)
case 2)
With a different version of the database, again, the upgrade, shutdown, and reconnect went fine.
(This database version works fine when upgrading from 1.8.1 to 2.2.9).
This time the startup went fine, but then:
java.sql.SQLException: org.hsqldb.HsqlException: java.io.UTFDataFormatException in statement [ ALTER TABLE aaa ADD xyz TINYINT DEFAULT 0]
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source)
(... our code ...)
Caused by: org.hsqldb.HsqlException: org.hsqldb.HsqlException: java.io.UTFDataFormatException
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
at org.hsqldb.persist.DataFileCache.get(Unknown Source)
at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
at org.hsqldb.index.NodeAVLDisk.findNode(Unknown Source)
at org.hsqldb.index.NodeAVLDisk.getLeft(Unknown Source)
at org.hsqldb.index.IndexAVL.firstRow(Unknown Source)
at org.hsqldb.persist.RowStoreAVL.rowIterator(Unknown Source)
at org.hsqldb.persist.RowStoreAVL.moveData(Unknown Source)
at org.hsqldb.TableWorks.moveData(Unknown Source)
at org.hsqldb.TableWorks.addColumn(Unknown Source)
at org.hsqldb.StatementSchema.getResult(Unknown Source)
at org.hsqldb.StatementSchema.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
... 16 more
Caused by: org.hsqldb.HsqlException: java.io.UTFDataFormatException
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
... 30 more
Caused by: java.io.UTFDataFormatException
at org.hsqldb.lib.StringConverter.readUTF(Unknown Source)
at org.hsqldb.lib.StringConverter.readUTF(Unknown Source)
at org.hsqldb.rowio.RowInputBinary.readString(Unknown Source)
at org.hsqldb.rowio.RowInputBinary.readChar(Unknown Source)
at org.hsqldb.rowio.RowInputBase.readData(Unknown Source)
at org.hsqldb.rowio.RowInputBinary.readData(Unknown Source)
at org.hsqldb.rowio.RowInputBase.readData(Unknown Source)
at org.hsqldb.rowio.RowInputBinary.readData(Unknown Source)
at org.hsqldb.rowio.RowInputBinaryDecode.readData(Unknown Source)
at org.hsqldb.RowAVLDisk.<init>(Unknown Source)
... 31 more
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Upgrading from 1.8.1 to 2.3.0 snapshot 20130206, and running into character encoding problems
and strange OOMs.
Not sure if there are encoding problems in our existing 1.8.1 database.
Is there a change in UTF-8 handling/checking in 2.3.0?
Is it possible that bad UTF-8 encoded LONGVARCHAR rows are causing a corrupt database after upgrade in case 1) below?
Is it necessary to find and correct or purge the database of badly encoded LONGVARCHAR entries
before or after upgrading to 2.3.0? How could we do that?
Thanks for your help.
Details:
Upgrading from 1.8.1 to 2.2.9 works fine, but that has a problem with creating lobs
due to this bug
so we are trying the snapshot.
We're doing SHUTDOWN COMPACT immediately after upgrading, as recommended.
We have a table, call it aaa, that contans a LONGVARCHAR column.
After upgrade, shutdown compact, and reconnecting, one of two things happens:
case 1)
OOM, even with -Xmx1024m on a 50 MB database, when hsqldb is processing the SET TABLE BBB INDEX line in the script file on reconnect.
where this line is the line after the SET TABLE AAA INDEX line in the script file.
case 2)
With a different version of the database, again, the upgrade, shutdown, and reconnect went fine.
(This database version works fine when upgrading from 1.8.1 to 2.2.9).
This time the startup went fine, but then:
If you send me the database, I will check and fix any issue. You can contact me via email (fredt at users.sourceforge.net).
Email sent. Thank you for your help with this. :)
A new snapshot dated 20130217 is available. Please check and report any issues.
http://www.hsqldb.org/repos/org/hsqldb/hsqldb/SNAPSHOT/
Please make sure the old version jar is not on your classpath.
Sorry for the delay in being able to check it out.
It appears that the snapshot resolved the problem. Thanks for your help. :)
Thanks for reporting.