Menu

#88 Use DataType.BLOB instead of DataType.LONGVARBINARY

closed
None
5
2008-07-29
2008-07-07
No

Some drivers do not handle DataType.LONGVARBINARY correctly, so it seems that it's better to use a standard DataType.BLOB type.
Attaching a patch.

Discussion

  • Roman Puchkovskiy

     
  • matthias g

    matthias g - 2008-07-07

    Logged In: YES
    user_id=1803108
    Originator: NO

    Hi Roman,

    I think that we might likely break the old API when we include this. Intuitively I would also expect that a BlobDataType would use a BLOB internally.
    I'm a not sure how to handle this since dbunit claims to be java 1.3 compatible. Let's see what the other opinions are out there.

    Regards,
    mat

     
  • matthias g

    matthias g - 2008-07-07

    Logged In: YES
    user_id=1803108
    Originator: NO

    Hi Roman,

    I think that we might likely break the old API when we include this. Intuitively I would also expect that a BlobDataType would use a BLOB internally.
    I'm a not sure how to handle this since dbunit claims to be java 1.3 compatible. Let's see what the other opinions are out there.

    Regards,
    mat

     
  • matthias g

    matthias g - 2008-07-07

    Logged In: YES
    user_id=1803108
    Originator: NO

    Hi Roman,

    I think that we might likely break the old API when we include this. Intuitively I would also expect that a BlobDataType would use a BLOB internally.
    I'm a not sure how to handle this since dbunit claims to be java 1.3 compatible. Let's see what the other opinions are out there.

    Regards,
    mat

     
  • Andrey Grebnev

    Andrey Grebnev - 2008-07-08

    Logged In: YES
    user_id=1170526
    Originator: NO

    In order to avoid problems with backward compatibility we can create some configuration parameter which will allow to switch between DataType.BLOB and DataType.LONGVARBINARY for BlobDataType class.

    I vote for this patch.

     
  • matthias g

    matthias g - 2008-07-08

    Logged In: YES
    user_id=1803108
    Originator: NO

    Hi Roman,

    Basically there is already such a configuration parameter to specify a whole datatypeFactory. Have a look at the "http://www.dbunit.org/properties/datatypeFactory" property at http://www.dbunit.org/properties.html

    I think what you reported is more like a bug where nobody can say for sure which jdbc drivers+versions will be negatively affected. From my point of view a dataType should use the same SQLtype for reading/writing values so I agree with you.

    One more thing: did you test writing a blob into the db using your patch? I am not sure if it will work. I would greatly appreciate a unit test.

    regards,
    mat

     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    Actually, our application (http://atleap.dev.java.net/) uses dbunit with this patch already about 3 years. I don't know why this patch was not submitted before, possibly because for some time the dbunit project seemed stall.
    We are using dbunit there to import/export data during the build and to create a backup of the whole DB and restore it from the backup. So it surely works (during the import data is written to BLOBs, of course).
    Recently, we've tested this and it worked under Mysql, Postgres, Oracle, Derby, Hsqldb... possibly I've forgot something.
    Without this patch, under Derby the following error occurs:
    java.sql.SQLDataException: An attempt was made to get a data value of type 'BLOB' from a data value of type 'LONGVARBINARY'.

     
  • matthias g

    matthias g - 2008-07-19

    Logged In: YES
    user_id=1803108
    Originator: NO

    Hi Roman,

    sorry for the long pause. I checked this with oracle and found that it does not work anymore. The exception - after applying your patch - is the following:

    java.lang.ClassCastException: [B cannot be cast to oracle.sql.BLOB
    at oracle.jdbc.driver.OraclePreparedStatement.setObjectCritical(OraclePreparedStatement.java:9130)
    at oracle.jdbc.driver.OraclePreparedStatement.setObjectInternal(OraclePreparedStatement.java:8749)
    at oracle.jdbc.driver.OraclePreparedStatement.setObject(OraclePreparedStatement.java:9222)
    at org.dbunit.dataset.datatype.BlobDataType.setSqlValue(BlobDataType.java:74)
    at org.dbunit.database.statement.SimplePreparedStatement.addValue(SimplePreparedStatement.java:73)
    at org.dbunit.database.statement.AutomaticPreparedBatchStatement.addValue(AutomaticPreparedBatchStatement.java:63)
    at org.dbunit.operation.AbstractBatchOperation.execute(AbstractBatchOperation.java:186)

    My test code is as follows:

    byte[] b = new byte[]{1, 2, 3, 4, 5, 6};
    DefaultTableMetaData metadata = new DefaultTableMetaData("lob_test_table", new Column[]{new Column("myblob",DataType.BLOB)});
    DefaultTable expectedTable = new DefaultTable(metadata);
    expectedTable.addRow(new Object[]{b});
    IDataSet dataSet = new DefaultDataSet(expectedTable);
    DatabaseOperation.INSERT.execute(dbunitConn, dataSet);

    I would appreciate if you could provide a little test that shows how your Oracle implementation works with the patch. Perhaps you are using the oracle datatype factory which has an own BLOB datatype?

    regards,
    mat

     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    We're using dbunit-2.2.3 with patch which enables topological sorting on export (but it doesn't matter as here import is concerned). The code in build.xml is like
    <dbunit driver="${hibernate.connection.driver_class}"
    supportBatchStatement="false"
    url="${hibernate.connection.url}"
    userid="${hibernate.connection.username}"
    schema="${dbunit.schema}"
    password="${hibernate.connection.password}"
    datatypeFactory="org.dbunit.ext.oracle.OracleDataTypeFactory"
    escapePattern="${dbunit.escapepattern}"
    >
    <operation type="CLEAN_INSERT" src="${file}" format="xml"/>
    </dbunit>
    As you can see, the datatype factory is org.dbunit.ext.oracle.OracleDataTypeFactory.
    Oracle is Oracle XE 10g Release 2 (10.2).

     
  • matthias g

    matthias g - 2008-07-21

    Logged In: YES
    user_id=1803108
    Originator: NO

    Thanks for this information. I assume that if you use the OracleDataTypeFactory you wouldn't even recognise your change to the BlobDataType. Just to be sure could you please change the BlobDataType back to "LONGVARBINARY" and retry the import? I think that it should still work (at least with oracle and your configuration shown below)

     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    Yes, CLEAN_INSERT works with Oracle+OracleDataTypeFactory even without this patch. As I understand, the patch was not inspired by Oracle-related problems.

     
  • matthias g

    matthias g - 2008-07-22

    Logged In: YES
    user_id=1803108
    Originator: NO

    For which RDBMS systems do you need your patch? If you could kindly provide a unit test that lets me reproduce your error I would like to commit your change immediately. I really believe we should have a proof for this behaviour having a test.
    Thanks a lot in advance,
    mat

     
  • matthias g

    matthias g - 2008-07-22
    • assigned_to: nobody --> gommma
     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    Without this patch, on Derby (embedded) import fails, with patch applied it works.
    I tried to switch test connection to Derby (I worked with dbunit-2.2.3). I edited profile.properties, switched dbunit.profile to derby (dbunit.profile = derby), and added configuration for it:
    dbunit.profile.derby.driverClass = org.apache.derby.jdbc.EmbeddedDriver
    dbunit.profile.derby.connectionUrl = jdbc\:derby\:D\:\\TMP\\derby\\derby_db;create\=true
    dbunit.profile.derby.schema = APP
    dbunit.profile.derby.user = APP
    dbunit.profile.derby.password = APP
    dbunit.profile.derby.unsupportedFeatures =
    I also added Derby to the POM:
    <dependency>
    <groupId>org.apache.derby</groupId>
    <artifactId>derby</artifactId>
    <version>10.4.1.3</version>
    <scope>test</scope>
    </dependency>
    But when I run the tests (mvn test), I see multiple errors like this one: "org.dbunit.dataset.NoSuchTableException: EMPTY_MULTITYPE_TABLE". I surely misconfigured something, but I have no idea how to fix this.
    Could you please give an advice how to make tests work with Derby?

     
  • matthias g

    matthias g - 2008-07-23

    Logged In: YES
    user_id=1803108
    Originator: NO

    Similar to the "src/test/org.dbunit.HypersonicEnvironment" you need to create a environment for Derby that initially creates all the needed DB tables (see HypsersonicEnvironment.executeDdlFile" method). The default environment (org.dbunit.DatabaseEnvironment) does not create any DDL objects. As you can see in the HypersonicEnvironment the script "src/sql/hypersonic.sql" is used to setup the database structures. It would be very nice if you could provide such an environment for Derby incl. the additional necessities (profile.properties, derby.sql DDL ...) as a patch (of course when you got it running ;-))

     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    OK, I've made a DerbyEnvironment, and I've copied SQL from Hypersonic one. Now I have the following during the tests: "java.sql.SQLException: Table/View 'TEST_TABLE' already exists in Schema 'APP'.". This happens even if I drop the database manually before running the tests. Looks like the script is run several times. How can this be worked-around?

     
  • matthias g

    matthias g - 2008-07-26

    Logged In: YES
    user_id=1803108
    Originator: NO

    I suppose that you have the static method "executeDdlFile" (similar to HypersonicEnvironment). You could simply start the debugger or add a System.out.println to see from where/how often the initialization method is invoked. A simple workaround (if the method is really invoked multiple times) would be to create a static field like "private static boolean dbInitialized" that holds the status of the db initialization. If it has been initialized once you can skip all subsequent initialization attempts.

     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    Thanks for the help with this. I've added derby-embedded testing, currently some tests still fail.
    Some of them are in InsertOperationTest:
    testExecuteForwardOnly(org.dbunit.operation.InsertOperationTest)
    testExecute(org.dbunit.operation.InsertOperationTest)
    testExecuteCaseInsensitive(org.dbunit.operation.InsertOperationTest)
    These are the ones that should be fixed by the patch which is discussed here. When I apply the patch, these three failures disappear.
    Another test which fails is "testColumnDataType(org.dbunit.database.DatabaseTableMetaDataTest)". It expects to see a VARBINARY type but sees BLOB, that's why it fails. The problem is that Derby does not support VARBINARY type, it supports BLOB which seems to have the same semantics. So I don't know what to do with this test, I think dbunit developers should make the decision.
    Other tests which fail are the following:
    testResolveOperationTypes(org.dbunit.ant.DbUnitTaskTest)
    testExportFull(org.dbunit.ant.DbUnitTaskTest)
    testExportPartial(org.dbunit.ant.DbUnitTaskTest)
    testExportFlat(org.dbunit.ant.DbUnitTaskTest)
    testExportFlatWithDocytpe(org.dbunit.ant.DbUnitTaskTest)
    testExportXml(org.dbunit.ant.DbUnitTaskTest)
    testExportCsv(org.dbunit.ant.DbUnitTaskTest)
    testExportQuery(org.dbunit.ant.DbUnitTaskTest)
    testExportWithQuerySet(org.dbunit.ant.DbUnitTaskTest)
    testWithReferenceQuerySet(org.dbunit.ant.DbUnitTaskTest)
    testExportQueryMixed(org.dbunit.ant.DbUnitTaskTest)
    They complain that some tables are missing (for instance, here's one of the error messages: "file:E:/svnhome/dbunit-2.2.3/src/xml/antTestBuildFile.xml:118: org.dbunit.dataset.NoSuchTableException: EMPTY_MULTITYPE_TABLE"). I don't now how to fix them, sorry :) Possibly you could help.

    I'm attaching the patch containing this Derby testing support as it already demonstrates the problem which is fixed by the patch discussed here. Please note that the active testing profile here is 'derbyembedded'.
    File Added: dbunit-derby-testing.patch

     
  • Roman Puchkovskiy

    Patch which adds Derby testing support (some tests still fail)

     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    I've fixed the problem with Ant tasks tests, it was enough to configure dbunit.profile.* keys in the profile.properties, sorry for my hasty comments.
    So it looks like the Derby testing support is ok: 4 tests fail, 3 of them are fixed with patch discussed here, and one of them should be (possibly) rewritten (I mean testColumnDataType(org.dbunit.database.DatabaseTableMetaDataTest)) as it will not work in Derby.

     
  • matthias g

    matthias g - 2008-07-27

    Logged In: YES
    user_id=1803108
    Originator: NO

    First of all thanks a lot for the fine work. I committed the things on the current trunk (rev. 754) and would like you to retest it. I did some slight modifications with respect to the deletion of the database. It is done in the DerbyEnvironment instead of the maven build file. The advantage is that the tests are still executable in the IDE like this. I noticed that some of the ant task tests fail when switching to derby - this will need some further investigation. As you said the testColumnDataType(org.dbunit.database.DatabaseTableMetaDataTest) does not work any more in this environment and I am not yet sure on how to fix this. Do you have any suggestions?

    When the current trunk is working for you feel free to close this tracker item.

    Regards,
    mat

     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    Thanks for applying patches.

    I've tested trunk, revision 755.
    It seems to be broken, as PrintWriter#append(String) method is unknown to Java 1.4. I changed 'append' in SQLHelper:239 to 'println', and then dbunit could be compiled and passed tests.
    Then I've switched testing to Derby (editing the profile.properties file). Of all tests, two failed:
    testCreation_UnknownTable(org.dbunit.database.DatabaseTableMetaDataTest)
    testColumnDataType(org.dbunit.database.DatabaseTableMetaDataTest)
    First of them fails because it wants a strict match of error message which contains schema; PUBLIC is expected, but for Derby it's APP by default. Seems that this will fail for other databases, too, for instance for MSSQL. Possibly, message should be checked no so strictly here.
    Another test fails because it expects VARBINARY column but gets BLOB. I saw a list of unsupported features in the profile.properties for each DB, maybe the same mechanism may be used here (to disable this check for Derby, for instance)?

    I've also tested this version of dbunit with our application (and its Ant tasks) on Derby. Everything works fine.

     
  • matthias g

    matthias g - 2008-07-28

    Logged In: YES
    user_id=1803108
    Originator: NO

    I committed some more changes in rev. 756 which should fix the both tests that failed as well as guaranteeing the JDK 1.4 compatibility again.
    Therefore it is now necessary that you set the environment variable "JAVA_1_4_HOME" which is used for the maven-compiler-plugin so that we can be sure that in the future we are always 100% 1.4 compatible.
    This is one more change since DbUnit claimed to be 1.3 compatible until now which obviously was not true for the last releases (I tried to compile against 1.3 and got more than 20 different compiler errors).

    So again I'd like to ask you to retest if everything works fine for you now.

    Thanks a lot for your valuable input!
    matthias

     
  • Roman Puchkovskiy

    • status: open --> closed
     
  • Roman Puchkovskiy

    Logged In: YES
    user_id=2015874
    Originator: YES

    I've tested trunk, revision 759, works fine. Thanks :)
    Closing this tracker item.