Menu
▾
▴
#12 Improve Hibernate Import Performance
Look into methods to improve performance of the
Hibernate back-end. Especially in import.
Need to find out why import of 1.6MB XTM file takes up
to an hour to process against a MySQL database. Could
be related to connection pooling or to the fine-grained
transactions used by the back-end. This may mean that
it is necessary to allow coarser-grained transactions
to be controlled at the API level. In addition need to
look into the processing of merges - perhaps deferring
merge processing to the end of the import.
Logged In: YES
user_id=176992
Can also avoid import overhead by altering XTMBuilder.
Currently it creates a BaseName and then incrementally adds
the name string and each scope theme, causing merge checks
each time.
Instead, the base name string and themes should be gathered
up and only turned into a base name when the first variant
or the end of the base name syntax is encountered.
Logged In: YES
user_id=176992
Relaxing the unique constraints on Locators gave a 20x speed
increase from 1.5 topics/second to 30 topics/second for test
data.
Logged In: YES
user_id=176992
It now looks like the 20x speedup was a fluke result...I
wish I could work out why it happened!
However, using Hibernate 2.0 features, it is possible to
index columns. So far, creating indexes to assist locator
and subject identity point lookups have increased import
time from 2.8 topics per second to 6.5 topics per second for
the opera-test topic map.
Logged In: YES
user_id=176992
Some minor speedups achieved through optimisation of indexes
and improved URL parsing code. However, it looks like a
thorough overhaul (maybe a complete rewrite) of the JDBC
backend may be required. So I'm putting this off now to post
0.9.0.