Re: [TM4J-users] RE: importing xtm to Db with hibernate

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi All,

After a bit of a break I'm having another look at tm4j. I tried to use this approach of loading an
entire existing topic map into my hibernate backend, using some of Daniel's code below, but I get
the error below from the line 

tm.getFactory().copy(importedTM);

If Kal or anyone else has any clues what is going wrong here I'd be grateful.

Thanks,

Richard.

Heres the stack trace:

Problem with checked-in Statement, discarding.
java.lang.NullPointerException
        at oracle.jdbc.dbaccess.DBData.clearItem(DBData.java:431)
        at oracle.jdbc.dbaccess.DBDataSetImpl.clearItem(DBDataSetImpl.java:3528)
        at
oracle.jdbc.driver.OraclePreparedStatement.clearParameters(OraclePreparedStatement.java:3401)
        at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:452)
        at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:138)
        at
com.mchange.v2.c3p0.impl.C3P0PooledConnection$1$WrapperStatementHelper.doClose(C3P0PooledConnection.java:429)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnection$2.close(C3P0PooledConnection.java:476)
        at net.sf.hibernate.impl.BatcherImpl.closePreparedStatement(BatcherImpl.java:262)
        at net.sf.hibernate.impl.BatcherImpl.closeStatement(BatcherImpl.java:136)
        at net.sf.hibernate.impl.BatcherImpl.executeBatch(BatcherImpl.java:125)
        at net.sf.hibernate.impl.BatcherImpl.prepareStatement(BatcherImpl.java:59)
        at net.sf.hibernate.impl.BatcherImpl.prepareStatement(BatcherImpl.java:56)
        at net.sf.hibernate.impl.BatcherImpl.prepareBatchStatement(BatcherImpl.java:109)
        at net.sf.hibernate.persister.EntityPersister.insert(EntityPersister.java:460)
        at net.sf.hibernate.persister.EntityPersister.insert(EntityPersister.java:442)
        at net.sf.hibernate.impl.ScheduledInsertion.execute(ScheduledInsertion.java:29)
        at net.sf.hibernate.impl.SessionImpl.executeAll(SessionImpl.java:2382)
        at net.sf.hibernate.impl.SessionImpl.execute(SessionImpl.java:2335)
        at net.sf.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1775)
        at net.sf.hibernate.impl.SessionImpl.getQueries(SessionImpl.java:1536)
        at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1501)
        at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1491)
        at org.tm4j.topicmap.hibernate.PersistenceWrapper.find(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicMapImpl.getObjectByID(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicMapFactoryImpl.checkID(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicMapFactoryImpl.save(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicMapFactoryImpl.createBaseName(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicImpl.createName(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicImpl.createName(Unknown Source)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:506)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:259)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:212)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:203)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:96)
        at mips.genre.topicmaps.Hibernate.main(Hibernate.java:67)
org.tm4j.topicmap.TopicMapRuntimeException: Storage Exception.
Cause: org.tm4j.topicmap.TopicMapRuntimeException: Storage exception.
Cause: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is
not in deathmarch.
Cause:Internal inconsistency: A (not new) checking-out statement is not in deathmarch.
Cause:Storage exception.
Cause: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is
not in deathmarch.
Cause:Internal inconsistency: A (not new) checking-out statement is not in deathmarch.
        at org.tm4j.topicmap.hibernate.TopicMapFactoryImpl.save(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicMapFactoryImpl.createBaseName(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicImpl.createName(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicImpl.createName(Unknown Source)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:506)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:259)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:212)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:203)
        at org.tm4j.topicmap.TopicMapFactoryBase.copy(TopicMapFactoryBase.java:96)
        at mips.genre.topicmaps.Hibernate.main(Hibernate.java:67)
Caused by: org.tm4j.topicmap.TopicMapRuntimeException: Storage exception.
Cause: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is
not in deathmarch.
Cause:Internal inconsistency: A (not new) checking-out statement is not in deathmarch.
        at org.tm4j.topicmap.hibernate.PersistenceWrapper.find(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicMapImpl.getObjectByID(Unknown Source)
        at org.tm4j.topicmap.hibernate.TopicMapFactoryImpl.checkID(Unknown Source)
        ... 10 more
Caused by: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement
is not in deathmarch.
        at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.undeathmarchStatement(GooGooStatementCache.java:437)
        at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.removeStatement(GooGooStatementCache.java:273)
        at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:146)
        at
com.mchange.v2.c3p0.impl.C3P0PooledConnection$1$WrapperStatementHelper.doClose(C3P0PooledConnection.java:429)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnection$2.close(C3P0PooledConnection.java:476)
        at net.sf.hibernate.impl.BatcherImpl.closePreparedStatement(BatcherImpl.java:262)
        at net.sf.hibernate.impl.BatcherImpl.closeStatement(BatcherImpl.java:136)
        at net.sf.hibernate.impl.BatcherImpl.executeBatch(BatcherImpl.java:125)
        at net.sf.hibernate.impl.BatcherImpl.prepareStatement(BatcherImpl.java:59)
        at net.sf.hibernate.impl.BatcherImpl.prepareStatement(BatcherImpl.java:56)
        at net.sf.hibernate.impl.BatcherImpl.prepareBatchStatement(BatcherImpl.java:109)
        at net.sf.hibernate.persister.EntityPersister.insert(EntityPersister.java:460)
        at net.sf.hibernate.persister.EntityPersister.insert(EntityPersister.java:442)
        at net.sf.hibernate.impl.ScheduledInsertion.execute(ScheduledInsertion.java:29)
        at net.sf.hibernate.impl.SessionImpl.executeAll(SessionImpl.java:2382)
        at net.sf.hibernate.impl.SessionImpl.execute(SessionImpl.java:2335)
        at net.sf.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1775)
        at net.sf.hibernate.impl.SessionImpl.getQueries(SessionImpl.java:1536)
        at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1501)
        at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1491)
        ... 13 more

--- Daniel Meier <mei...@hi...> wrote:

> 
> Hello Kal
> 
> Thx for your answer. As you said, I reimplemented the import.
> First I create a topic map in memory, then I copy the topic map
> to the database. This is working fine, expect the topic map has
> scopes defined. I tried the tmworld.xtm topic map and I get an
> java.lang.ClassCastException at
> org.tm4j.topicmap.hibernate.ScopedObjectImpl.setScope(ScopedObjectImpl.java:
> 223)
> 
> Do you have an idea why this happens? I use tm4j 0.9.7src.
> Hope you can help me...thx in advance...
> 
> My code looks like:
> 
> public class copyToDB {
> 	public static void main(String[] args) {
> 		new copyToDB(args);
> 	}
> 	public copyToDB(String[] args) {
> 		File f = new File("src/tm/tmworld.xtm");
> 		try {
> 			TopicMap importedTM = loadXtmFromFile(f);
> 						
> 			// Hibernate
> 			TopicMapProviderFactory tmpf = new
> org.tm4j.topicmap.hibernate.TopicMapProviderFactoryImpl();
> 			Properties providerProps = new Properties();
> 			providerProps.load(new
> FileInputStream("src/tm/provider.properties"));
> 			TopicMapProvider provider =
> tmpf.newTopicMapProvider(providerProps);
> 
> 			
> 			ProviderTransaction txn =
> provider.openTransaction();
> 
> 			Locator base =
> provider.getLocatorFactory().createLocator("URI", f.toURL().toString());
> 			TopicMap tm = provider.createTopicMap(base);
> 			tm.getFactory().copy(importedTM);
> 						
> 			txn.commit();
> 			
> 		} catch(Exception e) {
> 			e.printStackTrace();
> 		}
> 	}
> 	
> 	public static TopicMap loadXtmFromFile(File f) throws Exception {
> 	    if (!f.exists()) {
> 	    	throw new FileNotFoundException("Could not locate the
> specified input file.");
> 	    }
> 	    TopicMapProviderFactory m_providerFactory = new
> org.tm4j.topicmap.memory.TopicMapProviderFactoryImpl();
> 	    TopicMapProvider m_provider =
> m_providerFactory.newTopicMapProvider(System.getProperties());
> 	    Locator baseLocator =
> m_provider.getLocatorFactory().createLocator("URI", f.toURL().toString());
> 	    TopicMapSource src = new SerializedTopicMapSource(new
> FileInputStream(f), baseLocator);
> 	    
>             return m_provider.addTopicMap(src);
> 	}
> }
> 
> 
> 
> Hi Daniel,
> 
> Yeah its slow, and partially it is due to the frequency of commits that
> Hibernate makes, but there is also a more tricky fundamental problem. If you
> are importing an XTM file, there is no guarantee that the XTM is properly
> merged. That means that as you add topics you need to check to see if they
> should merge - which in turn means you need to check the subject indicators,
> source locators and scoped names of the topics that you have imported up
> until now. So the XTM import code uses the indexes to do those lookups. Of
> course, with the Hibernate implementation the indexes are implemented as
> queries - so that means that you do need to commit at least after every
> topic is added for merging to work properly.
> 
> So if you have a really large TM, or if you are importing an XTM file into
> an existing topic map in the database (i.e. merging a file into an existing
> TM), then I think this is always going to be a problem
> 
> However, your case is slightly different (and probably quite common) - you
> have a relatively large XTM file, but it is not so large that it could not
> all be processed for merging in memory first (you might a big chunk of
> memory assigned to the VM, but it should be doable). So perhaps you could
> make a tuned import function for the Hibernate backend that does an import
> into a temporary in-memory backend first, applies all the merging, and then
> writes the objects to Hibernate in a single transaction. My feeling is that
> you should be able to do all of this using the existing APIs - we added a
> feature for transaction control on the backends in the last version. 
> 
> In the long term, I think the fix for this is to be smarter about the import
> into the database and to have a separate importer for Hibernate backends
> that works with the database instead of against it as we have at the moment.
> But perhaps for your short term needs doing the two-step XTM -> Memory ->
> Hibernate import might work?
> 
> Cheers,
> 
> Kal
> 
> > -----Original Message-----
> > From: tm4...@li... [mailto:tm4j-users-
> > ad...@li...] On Behalf Of Daniel Meier
> > Sent: 12 October 2005 19:21
> > To: tm4...@li...
> > Subject: [TM4J-users] importing xtm to Db with hibernate
> > 
> > Hello
> > 
> > 
> > 
> > Im trying to import a "big" xtm file (~6MByte, ~4000 topics and ~ 4000
> > assocciations ) to an oracle 8 database (hibernate backend).
> > 
> > Code looks like:
> > 
> > 
> > 
> > TopicMapProviderFactory factory = TopicMapProviderFactory.newInstance();
> > 
> > TopicMapProvider provider = factory.newTopicMapProvider();
> > 
> > TopicMapSource source = new
> > SerializedTopicMapSource("c:\\temp\\opera.xtm");
> > 
> > TopicMap map = provider.addTopicMap(source);
> > 
> > 
> > 
> > 
> > 
> > Now this process takes very looong (>1h). When I log the hibernate sql
> > commands,
> > 
> > I see that the backend is autocommiting every insert/update. This is in my
> > mind
> > 
> > not good, and is probably the problem. One commit at the end is enough
> > during an import.
> > 
> > So I set autocommit in the provider.properties to false, and added the
> > code:
> > 
> > .
> > 
> > TopicMap map = provider.addTopicMap(source);
> > 
> > Provider.getOpenTransaction().commit();
> > 
> > 
> > 
> > But this doesen't helped.
> > 
> > Maybe the problem is that provider.isTransactional() returns false. I
> > think the oracle DB
> > 
> > should be able to handle transactions. (I use the jdbc driver from
> > oracle).
> > 
> > 
> > 
> > What can I do?
> > 
> > 
> > 
> > Greetings, Daniel
> 
> 
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads, discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> Tm4j-users mailing list
> Tm4...@li...
> https://lists.sourceforge.net/lists/listinfo/tm4j-users
> 

___________________________________________________________ 
Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com