matplotlib

Hello Clovis.
(Also, hello list; I <heart> matplotlib, thank you developers!)

On Sun, May 14, 2006 at 09:04:27AM -0300, clovis@... wrote:

> I followed up the 'memory leak' discussion in the sourceforge list
> and I know the Matplotlib-FAQ entry about this subject. I've also
> seen John Hunter's post about the need of matching figure/close pairs.
> Anyway, I still feel that there are problems in this subject, which
> can be exposed by the following script (for Windows, but can easily
> be adapted to Unix).
> As can be seen by the results (also given below), there is a steady
> increase in memory usage which is not recovered!
> Any clues????

Could some of your difficulties could be related to Python's ill memory
management? It keeps a death grip on certain types of allocated memory, 
never freeing it to the OS.  Evan Jones made a patch which 
helped that problem a bit.   His patch has been incorporated into 
Python 2.5:

http://evanjones.ca/python-memory.html

Joseph Sheedy
Technical Specialist, 3TIER Environmental Forecast Group
josephs@... | (206)325-1573 x116

>>>>> "clovis" == clovis  <clovis@...> writes:

    clovis> All, I followed up the 'memory leak' discussion in the
    clovis> sourceforge list and I know the Matplotlib-FAQ entry about
    clovis> this subject. I've also seen John Hunter's post about the
    clovis> need of matching figure/close pairs.  Anyway, I still feel
    clovis> that there are problems in this subject, which can be
    clovis> exposed by the following script (for Windows, but can
    clovis> easily be adapted to Unix).

    clovis> As can be seen by the results (also given below), there is
    clovis> a steady increase in memory usage which is not recovered!

If I recall correctly, there is a known leak in tkagg when you create
multiple canvases, and this is in Tk and not matplotlib proper.  Todd
may have something to add here.  

JDH

>
>
>>>>"clovis" == clovis  <clovis@...> writes:
>>>>        
>>>>
>
>    clovis> All, I followed up the 'memory leak' discussion in the
>    clovis> sourceforge list and I know the Matplotlib-FAQ entry about
>    clovis> this subject. I've also seen John Hunter's post about the
>    clovis> need of matching figure/close pairs.  Anyway, I still feel
>    clovis> that there are problems in this subject, which can be
>    clovis> exposed by the following script (for Windows, but can
>    clovis> easily be adapted to Unix).
>
>    clovis> As can be seen by the results (also given below), there is
>    clovis> a steady increase in memory usage which is not recovered!
>
>If I recall correctly, there is a known leak in tkagg when you create
>multiple canvases, and this is in Tk and not matplotlib proper.  Todd
>may have something to add here.  
>
>JDH
>
>
>  
>
Yesterday you mentioned that that 'memory leak' would probably be caused 
by Tkagg
and not by matplotlib. You also mentioned that it would be good to hear 
Todd Miller
about this subject (I agree).

Following your idea I tested the memory usage under different backends.
The results are given below:

##############
#TKAgg results
##############
Date/time of test   = Wed May 17 10:08:39 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = TkAgg
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False  SAVEFIG=True
Memory usage before/after figure[ 0] =    15740     21564      5824
Memory usage before/after figure[ 1] =    21564     25944      4380
Memory usage before/after figure[ 2] =    25944     30316      4372
Memory usage before/after figure[ 3] =    30316     34668      4352
Memory usage before/after figure[ 4] =    34668     39020      4352
Memory usage before/after figure[ 5] =    39020     43376      4356
Memory usage before/after figure[ 6] =    43376     47740      4364
Memory usage before/after figure[ 7] =    47740     52096      4356
Memory usage before/after figure[ 8] =    52096     56472      4376
Memory usage before/after figure[ 9] =    56472     60836      4364

############
#Agg results
############
Date/time of test   = Wed May 17 09:24:26 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = Agg
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False  SAVEFIG=True
Memory usage before/after figure[ 0] =    14160     17008      2848
Memory usage before/after figure[ 1] =    17008     17504       496
Memory usage before/after figure[ 2] =    17504     17604       100
Memory usage before/after figure[ 3] =    17604     17604         0
Memory usage before/after figure[ 4] =    17604     17588       -16
Memory usage before/after figure[ 5] =    17588     17608        20
Memory usage before/after figure[ 6] =    17608     17600        -8
Memory usage before/after figure[ 7] =    17600     17596        -4
Memory usage before/after figure[ 8] =    17596     17584       -12
Memory usage before/after figure[ 9] =    17584     17604        20

##############
#Cairo results
##############
Date/time of test   = Wed May 17 10:29:09 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = Cairo
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False  SAVEFIG=True
Memory usage before/after figure[ 0] =    14024     15436      1412
Memory usage before/after figure[ 1] =    15436     15944       508
Memory usage before/after figure[ 2] =    15944     16252       308
Memory usage before/after figure[ 3] =    16252     16460       208
Memory usage before/after figure[ 4] =    16460     16468         8
Memory usage before/after figure[ 5] =    16468     18448      1980
Memory usage before/after figure[ 6] =    18448     18464        16
Memory usage before/after figure[ 7] =    18464     16744     -1720
Memory usage before/after figure[ 8] =    16744     18144      1400
Memory usage before/after figure[ 9] =    18144     18488       344

#################
#GTKCairo results
#################
Date/time of test   = Wed May 17 10:28:45 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = GTKCairo
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False  SAVEFIG=True
Memory usage before/after figure[ 0] =    20888     26384      5496
Memory usage before/after figure[ 1] =    26384     26972       588
Memory usage before/after figure[ 2] =    26972     28224      1252
Memory usage before/after figure[ 3] =    28224     27848      -376
Memory usage before/after figure[ 4] =    27848     27440      -408
Memory usage before/after figure[ 5] =    27440     29444      2004
Memory usage before/after figure[ 6] =    29444     29668       224
Memory usage before/after figure[ 7] =    29668     28840      -828
Memory usage before/after figure[ 8] =    28840     28804       -36
Memory usage before/after figure[ 9] =    28804     29884      1080

############
#GTK results
############
Date/time of test   = Wed May 17 09:20:02 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = GTK
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False  SAVEFIG=True
Memory usage before/after figure[ 0] =    22128     28044      5916
Memory usage before/after figure[ 1] =    28048     28624       576
Memory usage before/after figure[ 2] =    28624     28804       180
Memory usage before/after figure[ 3] =    28804     28872        68
Memory usage before/after figure[ 4] =    28872     28948        76
Memory usage before/after figure[ 5] =    28948     29020        72
Memory usage before/after figure[ 6] =    29020     29080        60
Memory usage before/after figure[ 7] =    29080     29144        64
Memory usage before/after figure[ 8] =    29144     29240        96
Memory usage before/after figure[ 9] =    29240     29292        52

###############
#GTKAgg results
###############
Date/time of test   = Wed May 17 09:20:24 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = GTKAgg
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False  SAVEFIG=True
Memory usage before/after figure[ 0] =    23880     31124      7244
Memory usage before/after figure[ 1] =    31124     31708       584
Memory usage before/after figure[ 2] =    31708     31904       196
Memory usage before/after figure[ 3] =    31904     32308       404
Memory usage before/after figure[ 4] =    32308     32388        80
Memory usage before/after figure[ 5] =    32388     32144      -244
Memory usage before/after figure[ 6] =    32144     32560       416
Memory usage before/after figure[ 7] =    32560     32276      -284
Memory usage before/after figure[ 8] =    32276     32380       104
Memory usage before/after figure[ 9] =    32380     32444        64

############
#EMF results
############
Date/time of test   = Wed May 17 09:18:19 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = EMF
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=True  SAVEFIG=True
Memory usage before/after figure[ 0] =    12352     13268       916
Memory usage before/after figure[ 1] =    13268     13732       464
Memory usage before/after figure[ 2] =    13732     13896       164
Memory usage before/after figure[ 3] =    13896     13900         4
Memory usage before/after figure[ 4] =    13900     13804       -96
Memory usage before/after figure[ 5] =    13804     13904       100
Memory usage before/after figure[ 6] =    13904     13904         0
Memory usage before/after figure[ 7] =    13904     13912         8
Memory usage before/after figure[ 8] =    13912     13836       -76
Memory usage before/after figure[ 9] =    13836     13920        84

###########
#PS results
###########
Date/time of test   = Wed May 17 09:21:15 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = PS
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False  SAVEFIG=True
Memory usage before/after figure[ 0] =    12912     14032      1120
Memory usage before/after figure[ 1] =    14032     14508       476
Memory usage before/after figure[ 2] =    14508     16344      1836
Memory usage before/after figure[ 3] =    16344     16548       204
Memory usage before/after figure[ 4] =    16548     17032       484
Memory usage before/after figure[ 5] =    17032     16996       -36
Memory usage before/after figure[ 6] =    16996     16648      -348
Memory usage before/after figure[ 7] =    16648     14876     -1772
Memory usage before/after figure[ 8] =    14876     16928      2052
Memory usage before/after figure[ 9] =    16928     16804      -124

############
#SVG results
############
Date/time of test   = Wed May 17 09:21:34 2006
OS version          = 2.4.3 (#69, Apr 11 2006, 15:32:42) [MSC v.1310 32 
bit (Intel)]
OS platform         = win32
Matplotlib version  = 0.87
Matplotlib revision = $Revision: 1.122 $
Matplotlib backend  = SVG
Column #0 = figure index
Column #1 = memory usage before figure
Column #2 = memory usage after  figure
Column #3 = (after-before) memory

Configuration SHOWFIG=False  SAVEFIG=True
Memory usage before/after figure[ 0] =    12844     13648       804
Memory usage before/after figure[ 1] =    13648     14128       480
Memory usage before/after figure[ 2] =    14128     14224        96
Memory usage before/after figure[ 3] =    14224     14224         0
Memory usage before/after figure[ 4] =    14224     14224         0
Memory usage before/after figure[ 5] =    14224     14228         4
Memory usage before/after figure[ 6] =    14228     14228         0
Memory usage before/after figure[ 7] =    14228     14228         0
Memory usage before/after figure[ 8] =    14228     14228         0
Memory usage before/after figure[ 9] =    14228     14228         0

As can be seen by the previously show results, it seems that the TKAgg 
still has some problems.
I'm not worried the the 'DC level' of the backends or with the results 
concerning figure[0].
The problem I'm pointing is the continuous increase in memory usage in 
the TKAgg backend.
The script used for these tests is:

import pylab
import os
import time
N = 10              # number of loops to execute
SAVEFIG = True      # SAVEFIG execution flag
SHOWFIG = False     # SHOWFIG execution flag

report_filename = 'memory_report_%s.txt' % pylab.matplotlib.get_backend()
fid = file(report_filename,'wt')
fid.write('Date/time of test   = %s\n' % time.asctime())
fid.write('OS version          = %s\n' % os.sys.version)
fid.write('OS platform         = %s\n' % os.sys.platform)
fid.write('Matplotlib version  = %s\n' % pylab.matplotlib.__version__)
fid.write('Matplotlib revision = %s\n' % pylab.matplotlib.__revision__)
fid.write('Matplotlib backend  = %s\n' % pylab.matplotlib.get_backend())
fid.write('Column #0 = figure index\n')
fid.write('Column #1 = memory usage before figure\n')
fid.write('Column #2 = memory usage after  figure\n')
fid.write('Column #3 = (after-before) memory\n')
      
pylab.ion()
a=pylab.arange(0,10)

def report_memory():
    ### Attention: the path to the pslist utility should be adjusted 
according to installation!
    if os.sys.platform == 'win32':
        ps_exe_filename = os.path.join(os.getcwd(),'pslist.exe')    
#Build ps filename
        a = os.popen('%s -m python' % ps_exe_filename).readlines()  
#Build and execute command
        b = a[8]
        c = b.split()
        return int(c[3])
    else:
        print 'Sorry, you have to adapt the command for your OS!'
        return 0

def figureloop(N):
    for i in range(0,N):
        memory_usage_before = report_memory()
        fid.write('Memory usage before/after figure[%2d] = %8d' % (i, 
memory_usage_before))
        pylab.figure(i)
        pylab.plot(a,2*a)
        figurename = 'fig%02d' % i
        if SAVEFIG:
            pylab.savefig(figurename)
            pylab.savefig(figurename+'.eps')
        if SHOWFIG:
            pylab.show()
        pylab.close(i)
        time.sleep(1.0)                 # wait 1.0 second before 
inspecting memory usage
        if os.path.isfile(figurename):  # remove figure ...
            #os.remove(figurename)
            pass
        memory_usage_after = report_memory()
        delta_memory = memory_usage_after - memory_usage_before
        fid.write('  %8d  %8d\n' % (memory_usage_after, delta_memory))
        print '%2d %6d %6d %6d' % (i, memory_usage_before, 
memory_usage_after, delta_memory)
       

print 'Column #0 = figure index'
print 'Column #1 = memory usage before figure'
print 'Column #2 = memory usage after  figure'
print 'Column #3 = (after-before) memory'
print 'There is a sleep time of 1s between each figure!'
print 'Close Figure[0] to continue execution!'
print('\nConfiguration SHOWFIG=%s  SAVEFIG=%s' % (SHOWFIG, SAVEFIG))
fid.write('\nConfiguration SHOWFIG=%s  SAVEFIG=%s\n' % (SHOWFIG, SAVEFIG))
figureloop(N)

This script should be executed with the backend parameters, such as:
    python memory_test.py -dTKAgg
    python memory_test.py -dAgg
    etc
for all the desired backends.

Thanks for your support and congratulations for your great work 
(matplotlib).
clovis

After waiting for some days for some suggestion, I would like to ask if
there is some suggestion or "starting-point" for solving this issue.
Sincerely yours,
clovis

John Hunter wrote:

>>>>>>"clovis" == clovis  <clovis@...> writes:
>>>>>>            
>>>>>>
>
>    clovis> All, I followed up the 'memory leak' discussion in the
>    clovis> sourceforge list and I know the Matplotlib-FAQ entry about
>    clovis> this subject. I've also seen John Hunter's post about the
>    clovis> need of matching figure/close pairs.  Anyway, I still feel
>    clovis> that there are problems in this subject, which can be
>    clovis> exposed by the following script (for Windows, but can
>    clovis> easily be adapted to Unix).
>
>    clovis> As can be seen by the results (also given below), there is
>    clovis> a steady increase in memory usage which is not recovered!
>
>If I recall correctly, there is a known leak in tkagg when you create
>multiple canvases, and this is in Tk and not matplotlib proper.  Todd
>may have something to add here.  
>
>JDH
>
>
>-------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>Matplotlib-users mailing list
>Matplotlib-users@...
>https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>
>  
>

Hello 
I have encountered memory leak when using pylab.figure(), pylab.show(),
pylab.close(). I expected pylab.close() to flush the memory but this was not
the case. what am i doing wrong? Thanks in advance

Below is simple example
import os, sys, time
import gc
import matplotlib
matplotlib.use('TKAgg')
import pylab
import matplotlib.cbook as cbook

pid = os.getpid()
a2 = os.popen('ps -p %d -o vsz' % pid).readlines()
print 'memory before all figures the figure is drawn: ' + str(a2[1])

pylab.ion()
fig_list = []
for i in range(10):
    fig_list.append(pylab.figure())
    val = cbook.report_memory(i)
    print i, val

pylab.show()
for fig in fig_list:
    pylab.close(fig)

gc.collect()
print 'uncollectable list:', gc.garbage
a2 = os.popen('ps -p %d -o vsz' % pid).readlines()
print 'memory after all figures are closed : ' + str(a2[1])
******results******************************
memory before all figures the figure is drawn: 35424

0 4981
1 4981
2 5493
3 6005
4 6005
5 6517
6 6517
7 7029
8 7541
9 7541
uncollectable list: []
memory after all figures are closed : 80808
thanks
kwabena

-- 
View this message in context: http://old.nabble.com/Memory-leak-tp27741668p27741668.html
Sent from the matplotlib - users mailing list archive at Nabble.com.