Re: [TM4J-users] slow tm export

[Kal A. <ka...@te...>]

Hi Conal,

My guess is that the walk is just doing to many transactions. If you
don't have an "outer" transaction wrapping that walk, then each call to
the API will result in a separate transaction against the database -
which amounts to quite a lot of transactions to walk the whole map.

What I would recommend you try is creating a transaction to wrap around
the walk. If a transaction is already in progress, then the TM4J
Hibernate code will use that txn instead of creating one.

The code would be something like this:

ProviderTransaction txn = provider.getTransaction();
try {
  walker.walk();
} finally {
  txn.abort();
}

Hopefully that should give you a major increase in performance as it
will reduce the number of transactions used and allow client-side
caching done by Hibernate to be effective.

Cheers,

Kal

On Fri, 2004-09-10 at 06:38, Conal Tuohy wrote:
> I am developing an application which uses tm4j with Hibernate as the provider. 
> 
> The application is a website built with Cocoon. The topic map is harvested from a bunch of sources, and defines the information architecture of the site. Each page on the site represents a topic with all its associated topics and other characteristics. So each page requires querying for a few topics only. This works OK.
> 
> My problem is that I also want to be able to visualise the entire map, to check my harvesting/merging/etc; to do this I've exported the entire map, and it takes about an hour (on a 2.6GHz P4 with 2GB RAM). This seems like a long time to export 838kb of XTM code containing about 2800 topics, but only a few dozen associations. Is this to be expected? Or is there something peculiar in this? The MySQL database is local, and the export is definitely processor-bound. About 97% of the processor time is used by Tomcat (which is the container in which tm4j is running), and the remainder is used by MySQL. 
> 
> Some people have reported very slow performance with IMPORTING into hibernate - I haven't noticed that. Certainly it takes only a minute or so to harvest all these topics (which includes transforming them from other XML formats into XTM) and insert them into tm4j.
> 
> I see now that it's possible to configure TMNav to use the hibernate provider - so maybe I don't need to do this export at all. I don't know if this will speed up the visualisation - I'm certainly going to try that, but I'm still keen to hear if someone can tell me I'm doing something wrong - especially if it's something small and easy to fix :-) 
> 
> I'm appending the JavaScript code which does the export, calling Java classes to do the actual work. This code runs inside of Cocoon, as a "flowscript".
> 
> Thanks
> 
> Con
> 
> 
> --
> Conal Tuohy
> Senior Programmer
> New Zealand Electronic Text Centre
> www.nzetc.org
> 
> JavaScript code:
> 
> function export_map() {
> 	/* this program exports the "nzetc" topicmap to an XTM file, and then calls a Cocoon pipeline to display that XTM file */ 
> 
> 	/* this file to export into */
> 	var export_filename = cocoon.parameters["filename"];
> 	
> 	/* get the topic map from the topic map manager */
> 	var TopicMapManager = cocoon.getComponent("org.tm4j.tm4web.cocoon.TopicMapManagerComponent");
> 	var map = TopicMapManager.getTopicMap("nzetc");
> 	cocoon.releaseComponent(TopicMapManager);
> 
> 	/* write the topic map to a file */
> 	var walker = new Packages.org.tm4j.topicmap.utils.TopicMapWalker();
> 	var writer = new Packages.org.tm4j.topicmap.utils.XTMWriter();
> 	var file = new java.io.File(export_filename);
> 	var os = new java.io.FileOutputStream(file);
> 	var of = new Packages.org.apache.xml.serialize.OutputFormat();
> 	of.setEncoding("UTF-8");
> 	of.setIndenting(true);
> 	of.setIndent(2);
> 	var serializer = new Packages.org.apache.xml.serialize.XMLSerializer(os, of);
> 	walker.setHandler(writer);
> 	writer.setContentHandler(serializer);
> 	walker.walk(map);
> 	cocoon.sendPage("export/nzetc.xtm");
> }
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM. 
> Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Tm4j-users mailing list
> Tm4...@li...
> https://lists.sourceforge.net/lists/listinfo/tm4j-users
-- 
Kal Ahmed <ka...@te...>
techquila