#918 [SQL] Use Sparevector instead of std::vector

Completed
closed
Neil Hodgson
Scintilla (356)
5
2012-06-02
2012-03-22
Jérôme LAFORGE
No

Please find this patch where SpareVector is use instead of std::vector for memory usage optimization.

Discussion

  • There is a problem with upload file on sourceforge.
    So I use pastebin :
    http://pastebin.com/De4ETGme

     
  • Please find this new patch where I fix typo into commit message.
    http://pastebin.com/S3KavP3i

     
  • Neil Hodgson
    Neil Hodgson
    2012-03-22

    • assigned_to: nobody --> nyamatongwe
     
  • Neil Hodgson
    Neil Hodgson
    2012-03-22

    Have you measured memory usage on some example files to be sure this is a win?

     
  • I've done some tests with libmemusage.so and huge SQL file with 1 million same insert lines "INSERT INTO Store_Information (store_name, Sales, Date) VALUES ('Los Angeles', 900, 'Jan-10-1999')"

    For 1 million lines
    heap total with vector : 276461294
    heap total with SpareState : 266998740
    heap total with no file : 7152412

    detail of test (I've also attach report in text file)
    export LD_PRELOAD=/lib/libmemusage.so
    SciTE test.sql

    First test with no file opened :
    Memory usage summary: heap total: 7152412, heap peak: 1655120, stack peak: 313496
    total calls total memory failed calls
    malloc| 70767 5671948 0
    realloc| 13650 613784 0 (nomove:3982, dec:192, free:0)
    calloc| 4321 866680 0
    free| 75712 5920498
    ***************
    [vector sqlStatement] SQL file with 1 million of lines :
    Memory usage summary: heap total: 276461294, heap peak: 210477334, stack peak: 313496
    total calls total memory failed calls
    malloc| 272904 273215662 0
    realloc| 44641 2162931 0 (nomove:18832, dec:196, free:0)
    calloc| 19413 1082701 0
    free| 309446 274142890
    ***************
    [SpareState sqlStatement] SQL file with 1 million of lines :
    Memory usage summary: heap total: 266998740, heap peak: 210498922, stack peak: 313496
    total calls total memory failed calls
    malloc| 227018 264597424 0
    realloc| 29399 1410488 0 (nomove:11372, dec:204, free:0)
    calloc| 11824 990828 0
    free| 246773 56958118

     
  •  
    Attachments
  • Surely, I wasn't clear enough in my previous message, the gain isn't only for heap usage (diff ~ 3%).
    As, you can see, there is much more memory management with vector than SpareState (ex: malloc: +20%, realloc: +51%, calloc: +64%, free: +25%).
    This is due to vector reallocation.

     
  • Neil Hodgson
    Neil Hodgson
    2012-03-25

    When run with the example data lines, the vector stayed at size 0 since every line had state 0. To provoke the vector into expanding a merge statement can be added so that at least one line has non-zero state. It looks like the SpareState does help when that is done.

    Committed.

     
  • Neil Hodgson
    Neil Hodgson
    2012-03-25

    • milestone: --> Completed
     
  • You're right, I forgot to precise this point into the test file.
    thx for commit.

     
  • Another point, is it possible to add include directive for vector and algorithm directly SpareState.h. Thus, you can remove #include <vector> and #include <algorithm> from LexSQL.cxx (and others files).

     
  • Neil Hodgson
    Neil Hodgson
    2012-03-25

    Do not hide unnecessary #includes inside headers as it just makes compilation take longer and makes it harder to understand what is used.

     
  • Neil Hodgson
    Neil Hodgson
    2012-06-02

    • status: open --> closed