[TM4J-users] RE: importing xtm to Db with hibernate

[Daniel M. <mei...@hi...>]

Hello Kal

Thx for your answer. As you said, I reimplemented the import.
First I create a topic map in memory, then I copy the topic map
to the database. This is working fine, expect the topic map has
scopes defined. I tried the tmworld.xtm topic map and I get an
java.lang.ClassCastException at
org.tm4j.topicmap.hibernate.ScopedObjectImpl.setScope(ScopedObjectImpl.java:
223)

Do you have an idea why this happens? I use tm4j 0.9.7src.
Hope you can help me...thx in advance...

My code looks like:

public class copyToDB {
	public static void main(String[] args) {
		new copyToDB(args);
	}
	public copyToDB(String[] args) {
		File f = new File("src/tm/tmworld.xtm");
		try {
			TopicMap importedTM = loadXtmFromFile(f);
						
			// Hibernate
			TopicMapProviderFactory tmpf = new
org.tm4j.topicmap.hibernate.TopicMapProviderFactoryImpl();
			Properties providerProps = new Properties();
			providerProps.load(new
FileInputStream("src/tm/provider.properties"));
			TopicMapProvider provider =
tmpf.newTopicMapProvider(providerProps);

			
			ProviderTransaction txn =
provider.openTransaction();

			Locator base =
provider.getLocatorFactory().createLocator("URI", f.toURL().toString());
			TopicMap tm = provider.createTopicMap(base);
			tm.getFactory().copy(importedTM);
						
			txn.commit();
			
		} catch(Exception e) {
			e.printStackTrace();
		}
	}
	
	public static TopicMap loadXtmFromFile(File f) throws Exception {
	    if (!f.exists()) {
	    	throw new FileNotFoundException("Could not locate the
specified input file.");
	    }
	    TopicMapProviderFactory m_providerFactory = new
org.tm4j.topicmap.memory.TopicMapProviderFactoryImpl();
	    TopicMapProvider m_provider =
m_providerFactory.newTopicMapProvider(System.getProperties());
	    Locator baseLocator =
m_provider.getLocatorFactory().createLocator("URI", f.toURL().toString());
	    TopicMapSource src = new SerializedTopicMapSource(new
FileInputStream(f), baseLocator);
	    
            return m_provider.addTopicMap(src);
	}
}



Hi Daniel,

Yeah its slow, and partially it is due to the frequency of commits that
Hibernate makes, but there is also a more tricky fundamental problem. If you
are importing an XTM file, there is no guarantee that the XTM is properly
merged. That means that as you add topics you need to check to see if they
should merge - which in turn means you need to check the subject indicators,
source locators and scoped names of the topics that you have imported up
until now. So the XTM import code uses the indexes to do those lookups. Of
course, with the Hibernate implementation the indexes are implemented as
queries - so that means that you do need to commit at least after every
topic is added for merging to work properly.

So if you have a really large TM, or if you are importing an XTM file into
an existing topic map in the database (i.e. merging a file into an existing
TM), then I think this is always going to be a problem

However, your case is slightly different (and probably quite common) - you
have a relatively large XTM file, but it is not so large that it could not
all be processed for merging in memory first (you might a big chunk of
memory assigned to the VM, but it should be doable). So perhaps you could
make a tuned import function for the Hibernate backend that does an import
into a temporary in-memory backend first, applies all the merging, and then
writes the objects to Hibernate in a single transaction. My feeling is that
you should be able to do all of this using the existing APIs - we added a
feature for transaction control on the backends in the last version. 

In the long term, I think the fix for this is to be smarter about the import
into the database and to have a separate importer for Hibernate backends
that works with the database instead of against it as we have at the moment.
But perhaps for your short term needs doing the two-step XTM -> Memory ->
Hibernate import might work?

Cheers,

Kal

> -----Original Message-----
> From: tm4...@li... [mailto:tm4j-users-
> ad...@li...] On Behalf Of Daniel Meier
> Sent: 12 October 2005 19:21
> To: tm4...@li...
> Subject: [TM4J-users] importing xtm to Db with hibernate
> 
> Hello
> 
> 
> 
> Im trying to import a "big" xtm file (~6MByte, ~4000 topics and ~ 4000
> assocciations ) to an oracle 8 database (hibernate backend).
> 
> Code looks like:
> 
> 
> 
> TopicMapProviderFactory factory = TopicMapProviderFactory.newInstance();
> 
> TopicMapProvider provider = factory.newTopicMapProvider();
> 
> TopicMapSource source = new
> SerializedTopicMapSource("c:\\temp\\opera.xtm");
> 
> TopicMap map = provider.addTopicMap(source);
> 
> 
> 
> 
> 
> Now this process takes very looong (>1h). When I log the hibernate sql
> commands,
> 
> I see that the backend is autocommiting every insert/update. This is in my
> mind
> 
> not good, and is probably the problem. One commit at the end is enough
> during an import.
> 
> So I set autocommit in the provider.properties to false, and added the
> code:
> 
> .
> 
> TopicMap map = provider.addTopicMap(source);
> 
> Provider.getOpenTransaction().commit();
> 
> 
> 
> But this doesen't helped.
> 
> Maybe the problem is that provider.isTransactional() returns false. I
> think the oracle DB
> 
> should be able to handle transactions. (I use the jdbc driver from
> oracle).
> 
> 
> 
> What can I do?
> 
> 
> 
> Greetings, Daniel