#500 Opening large textfile consumes huge amout of memory

Squirrel SQL
open
nobody
Core (462)
5
2015-01-08
2007-03-23
bkoos
No

during the process of opening a large sql file (~50 MB)
the memory consumption increases > 1 GB, so opening a 300MB-file becomes impossible. why does the editor use so much memory. once the file is loaded, the memory usage goes down again to a reasonable amount.

Discussion

  • Rob Manning

    Rob Manning - 2007-03-24

    Logged In: YES
    user_id=1287991
    Originator: NO

    In the case of the NetBeans editor (syntax plugin) it is the following class which appears to allocate the most objects:

    org.netbeans.editor.BaseDocument.insertString(int, String, AttributeSet)

    Perhaps if we updated the netbeans jars, then maybe we would get some efficiency gains? I'm not the expert in this area.

    In the case of plain editor when we run out of memory the stack trace looks like:

    java.lang.OutOfMemoryError: Java heap space
    at javax.swing.text.GapContent.allocateArray(GapContent.java:77)
    at javax.swing.text.GapVector.resize(GapVector.java:197)
    at javax.swing.text.GapVector.shiftEnd(GapVector.java:212)
    at javax.swing.text.GapContent.shiftEnd(GapContent.java:328)
    at javax.swing.text.GapVector.open(GapVector.java:184)
    at javax.swing.text.GapVector.replace(GapVector.java:125)
    at javax.swing.text.GapContent.insertString(GapContent.java:115)
    at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:709)
    at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:693)
    at javax.swing.text.PlainDocument.insertString(PlainDocument.java:114)
    at javax.swing.JTextArea.append(JTextArea.java:470)
    at net.sourceforge.squirrel_sql.client.session.DefaultSQLEntryPanel.appendText(DefaultSQLEntryPanel.java:170)
    at net.sourceforge.squirrel_sql.client.session.SQLPanelAPI.appendSQLScript(SQLPanelAPI.java:430)
    at net.sourceforge.squirrel_sql.client.session.FileManager.loadScript(FileManager.java:106)
    at net.sourceforge.squirrel_sql.client.session.FileManager.open(FileManager.java:84)
    at net.sourceforge.squirrel_sql.client.session.SQLPanelAPI.fileOpen(SQLPanelAPI.java:202)
    at net.sourceforge.squirrel_sql.client.session.action.FileOpenAction.actionPerformed(FileOpenAction.java:21)
    at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
    ...

    Perhaps we are not using these efficiently, but if so I cannot tell. We are grabbing all of the lines of the file, sticking them into a StringBuffer, then putting the resulting string into the JTextArea / Document. Anyone have any ideas on how we could more efficiently use the JTextArea / Document? I saw a post that indicated that the massive memory footprint could be somehow related to registering as an UndoListener. If that is true, than perhaps we could register for undo events after loading the file and not before. Just thinking out loud here, I don't really have the answer on this one.

    Rob

     
  • Rob Manning

    Rob Manning - 2007-03-24
    • labels: 465987 --> 448694
     
  • Rob Manning

    Rob Manning - 2007-03-24

    Logged In: YES
    user_id=1287991
    Originator: NO

    One thing I tried (unsuccessfully) was to segment the script into 1MB chunks and then instead of appending to the JTextArea directly, I tried creating a new PlainDocument, then appending the chunks, and finally setting the document. This should have eliminated all of the view side-effects of changing the content of the document in JTextArea. It seems that GapContent.insertString makes an array copy of the string sent in, so at the very least you need twice the memory to insert any string. Even still, I can't account for the fact that the PlainDocument seems to be allocating about 10 times the amount of memory required by the content of the script. (I used a script that had about 20 million characters in it. At least that was the length of the string that was being appended. However, SQuirreL needed to allocate about 350 MB to handle PlainDocument.insertString with the script contents. The script size was approximately 30 MB.) I would never have guessed that PlainDocument would require this amount of memory, and I'm still stumped as to why it is the case. If anyone else has any ideas I'm all ears. The JVM I was using is 1.6.0-b105.

    Rob

     
  • Rob Manning

    Rob Manning - 2007-03-24
    • labels: 448694 --> Core
    • milestone: --> All
    • assigned_to: joco01 --> manningr
     
  • Rob Manning

    Rob Manning - 2007-03-25

    Logged In: YES
    user_id=1287991
    Originator: NO

    If you use Oracle, and you only want to execute the script without editing it, you should be able to do this by creating a script with the line:

    @<path to large script here>

    So, if you are using windows and your script is in c:\tmp\big_script.sql then execute this in the editor:

    @c:\tmp\big_script.sql

    Rob

     
  • Rob Manning

    Rob Manning - 2007-03-25

    Logged In: YES
    user_id=1287991
    Originator: NO

    I'm getting out of the way now, in case someone else wants to tackle this beast.

    I'm going to go lick my wounds now :(

    Rob

     
  • Rob Manning

    Rob Manning - 2007-03-25
    • assigned_to: manningr --> nobody
     
  • Rob Manning

    Rob Manning - 2007-04-03

    Logged In: YES
    user_id=1287991
    Originator: NO

    Both of these bugs have something in common:

    1686698 (Opening large textfile consumes huge amout of memory)

    1366016 (Import binary file causes hang)

    They both are caused by many calls to methods in GapContent
    as a result of loading data or reading data into/from a JTextArea.

    1366016: GapContent.getChars
    1686698: GapContent.insertString

    We would do well to reduce the dependence on this Java API implementation
    class or remove it from the call chain entirely (but how?).

    Rob

     
  • LeO

    LeO - 2009-11-27

    Hm' I am not pretty sure if this problem itself is fixed, cause a 17MB sql-file can be opened and the mem-consumption remains at approx. 120MB Heap-usage.

    BUT the Editor itself remains unresponsive, ie. edting is not possible at all. Tested with version 20090926. Is it worth to open a new Bug?

     

Log in to post a comment.