#1102 Memory Leak when using Replace function.
When using Geany on Windows 7 box with 8 core Intel processor (could not verify the processor name) and 32 GB of memory, there appears to be a memory leak or other similar problem.
I am using an OpenSUSE 13.2 virtual machine running in VirtualBox for comparisons. The Linux VM has 4GB of memory but only 2GB of swap memory because of the way I originally built the machine.
I am loading a 300MB text file that is a dump of a database table. The table has three columns separated by commas. Text values are double-quote delimited. For unknown reasons there is a \r\r\n character combination as the line ending for each row.
Geany is being used to:
1) Convert the double-quotes into pipe symbols where they are used as delimiters but not where they appear as the inch symbol in text fields.
2) Remove one of the \r characters from the \r\r\n character sequence.
The first thing I notice upon opening the 300MB file is that Geany uses 900MB to 900MB of memory.
Next, when I run the replace function to go from "\r\r\n" to |\r\n|, in Windows the memory usage gradually increases to around 1.3 to 1.5 GB (there is about 28 GB of available physical memory). In Linux it almost immediately jumped to 2.2 GB and then increased by about 100 MB per every two or so minutes. It topped out at 2.7 GB. Upon completion of the replace function and saving the file, the memory usage of Geany drops to about exactly 1 GB of memory in Windows. In Linux, memory usage by Geany stays at 2.7 GB.
I then start the second replacement operation to convert "," to |,|. Geany begins consuming more memory. In Windows, it's not much but Geany then crashes with a C++ runtime error that contains no error number or other useful information. Additionally, I had to break one of the files into 4 parts to perform the replacements. I then had to use Windows Notepad to reassemble the file because there is apparently a similar memory leak when using the Clipboard in Geany. In Linux, physical memory usage jumped to 3.5 GB and then 1.8 GB of swap memory was used up. In Linux, memory usage continues to increase until there is no more memory left. At that point, Geany closes without any error or warning messages.
If I close and open Geany between replacement operations, memory is released and I can work through the replacement process. For smaller files, the symptoms are the same but it takes longer to use up all available memory, meaning I can perform more replacement operations on several files before memory is eventually depleted.
I hope this can be fixed because Geany perform significantly better than the other applications I tried (MS Word, Notepad and Programmers Notepad). But, the files I have are only going to continue growing larger so eventually, Geany will no longer be an option.
Here's an addition to the above description. There are times during the replacement operation where memory runs out and the replace operation stops without Geany crashing. When you scan through the file you can find out where the replacement was at when the memory ran out because the replacements stopped in mid-file. This symptom is the same in both Windows and Linux.
Last edit: Patrick Headley 2015-06-25
This looks pretty much as expected based on the implementation of
Scintilla, the editing component Geany uses.
On 26 June 2015 at 09:27, Patrick Headley linxpatrick@users.sf.net wrote:
The buffer has to contain style information as well as the text, that
at least doubles the size, also there is a per-line overhead, which
will be doubled due to your \r\r\n sequences. So 900Mb including
Geany sounds quite plausible.
Your edits increase the size of the contents, so at some point the
buffer will be re-allocated larger, although Scintilla returns the
memory to the C++ library, that often doesn't return it to the OS, and
the behaviour will differ between versions of the library, which is
why the effects are different on Linux and windows.
To avoid problems with overlapping occurrences, all the occurrances of
the text that is going to be replaced are found and recorded first,
before any actual replacements happen. Clearly that will use memory
as it records them. That memory is returned to the C library as the
replacements occur AFAICT, but again that library may not return it to
the OS.
Unfortunately running out of memory is a deadly error, applications
can't do much about it.
If you actually ran Linux out of memory the OOM killer will kick in,
and kill a process to get memory back. As geany is the largest memory
user at that point, its likely that it is the one killed. The process
has no say in that matter.
That agrees with the analysis above.
Yes, some memory will be re-used between files (presuming you close
them after the replace) but it won't be perfect because of
fragmentation. Other uses of memory, such as during saving, may take
part of a previously released buffer block, meaning it is no longer
large enough to be re-used on the next file, so a new block is
allocated, and so memory grows.
Whilst we may find some minor leaks, I doubt that the reasons outlined
above for why memory grows are not going to change anytime soon.
Related
Bugs: #1102
After performing the same tests with Eclipse and seeing similar results I am satisfied with your response except for one thing. With Geany on Linux and Eclipse on Windows, memory usage appears to increase to the maximum available memory while Geany on Windows 7 doesn't exceed much more than 1GB. Is that a bug?
On 26 June 2015 at 11:22, Patrick Headley linxpatrick@users.sf.net wrote:
That again looks like different runtime libraries having different
policies about how much memory they take from the OS at a time and
about returning memory to the OS, noting that Eclipse brings the Java
runtime into the picture as well as C and C++ libraries used by Geany.
Related
Bugs: #1102