[Tm4j-developers] [ tm4j-Feature Requests-704521 ] Improve Hibernate Import Performance

[SourceForge.net <no...@so...>]

Feature Requests item #704521, was opened at 2003-03-16 15:22
Message generated for change (Settings changed) made by kal_ahmed
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=391882&aid=704521&group_id=27895

>Category: Performance Enhancement
>Group: Next Release (example)
Status: Open
Priority: 5
Submitted By: Kal Ahmed (kal_ahmed)
Assigned to: Nobody/Anonymous (nobody)
Summary: Improve Hibernate Import Performance

Initial Comment:
Look into methods to improve performance of the
Hibernate back-end. Especially in import.

Need to find out why import of 1.6MB XTM file takes up
to an hour to process against a MySQL database. Could
be related to connection pooling or to the fine-grained
transactions used by the back-end. This may mean that
it is necessary to allow coarser-grained transactions
to be controlled at the API level. In addition need to
look into the processing of merges - perhaps deferring
merge processing to the end of the import.

----------------------------------------------------------------------

Comment By: Kal Ahmed (kal_ahmed)
Date: 2003-08-24 17:22

Message:
Logged In: YES 
user_id=176992

It now looks like the 20x speedup was a fluke result...I
wish I could work out why it happened!

However, using Hibernate 2.0 features, it is possible to
index columns. So far, creating indexes to assist locator
and subject identity point lookups have increased import
time from 2.8 topics per second to 6.5 topics per second for
the opera-test topic map.

----------------------------------------------------------------------

Comment By: Kal Ahmed (kal_ahmed)
Date: 2003-08-23 08:50

Message:
Logged In: YES 
user_id=176992

Relaxing the unique constraints on Locators gave a 20x speed
increase from 1.5 topics/second to 30 topics/second for test
data.

----------------------------------------------------------------------

Comment By: Kal Ahmed (kal_ahmed)
Date: 2003-08-16 18:48

Message:
Logged In: YES 
user_id=176992

Can also avoid import overhead by altering XTMBuilder.
Currently it creates a BaseName and then incrementally adds
the name string and each scope theme, causing merge checks
each time.
Instead, the base name string and themes should be gathered
up and only turned into a base name when the first variant
or the end of the base name syntax is encountered.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=391882&aid=704521&group_id=27895