Menu

#12 Improve Hibernate Import Performance

After Next Release
open
nobody
5
2004-02-17
2003-03-16
Kal Ahmed
No

Look into methods to improve performance of the
Hibernate back-end. Especially in import.

Need to find out why import of 1.6MB XTM file takes up
to an hour to process against a MySQL database. Could
be related to connection pooling or to the fine-grained
transactions used by the back-end. This may mean that
it is necessary to allow coarser-grained transactions
to be controlled at the API level. In addition need to
look into the processing of merges - perhaps deferring
merge processing to the end of the import.

Discussion

  • Kal Ahmed

    Kal Ahmed - 2003-08-16

    Logged In: YES
    user_id=176992

    Can also avoid import overhead by altering XTMBuilder.
    Currently it creates a BaseName and then incrementally adds
    the name string and each scope theme, causing merge checks
    each time.
    Instead, the base name string and themes should be gathered
    up and only turned into a base name when the first variant
    or the end of the base name syntax is encountered.

     
  • Kal Ahmed

    Kal Ahmed - 2003-08-23

    Logged In: YES
    user_id=176992

    Relaxing the unique constraints on Locators gave a 20x speed
    increase from 1.5 topics/second to 30 topics/second for test
    data.

     
  • Kal Ahmed

    Kal Ahmed - 2003-08-24

    Logged In: YES
    user_id=176992

    It now looks like the 20x speedup was a fluke result...I
    wish I could work out why it happened!

    However, using Hibernate 2.0 features, it is possible to
    index columns. So far, creating indexes to assist locator
    and subject identity point lookups have increased import
    time from 2.8 topics per second to 6.5 topics per second for
    the opera-test topic map.

     
  • Kal Ahmed

    Kal Ahmed - 2004-01-31
    • labels: --> Performance Enhancement
    • milestone: --> 124586
     
  • Kal Ahmed

    Kal Ahmed - 2004-02-17
    • milestone: 124586 --> After Next Release
     
  • Kal Ahmed

    Kal Ahmed - 2004-02-17

    Logged In: YES
    user_id=176992

    Some minor speedups achieved through optimisation of indexes
    and improved URL parsing code. However, it looks like a
    thorough overhaul (maybe a complete rewrite) of the JDBC
    backend may be required. So I'm putting this off now to post
    0.9.0.

     

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.