Re: [Matplotlib-users] hist() memory leak?

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

  I *am* able to see the same leak in 0.90.1 using Agg, Pdg, Ps, and
Svg with the same memory amount being leaked each time, so its
probably in matplotlib code, although I do not have the SVN sources to
check it there to see if it has been fixed there.  I was originally
using the wxAgg backend, but during the batch job really all i needed
was the actual histogram numbers, so I looked through the code and
found matplotlib.mlab.hist which is all I really need, and doesn't
leak ;-).

Thanks for the help,
Luke Robison

On 8/9/07, Michael Droettboom <md...@st...> wrote:
> There have been a number of memory leaks resolved since the 0.90.1
> release.  However, there are still known memory leaks in all of the GUI
> backends, some of which are unfortunately just beyond easy reach of
> matplotlib.  If this is an automated process and you only care about the
> file output, you could try using the "Agg", "Pdf", "Ps" or "Svg"
> backends, e.g.:
>
>         import matplotlib
>         matplotlib.use("Agg")
>
> I tried your script with both 0.90.1 and the latest svn, and I could
> reproduce your leak with the TkAgg backend, but not with the Agg backend.
>
> If you need a GUI, you may want to try using the latest svn version if
> you can.  The leaks still exist there, but they are much smaller.
>
> BTW -- you can get the version of matplotlib with:
>
>  >>> import matplotlib
>  >>> matplotlib.__version__
> '0.90.1'
>
> Cheers,
> Mike
>
>
> Luke Robison wrote:
> > I'm writing a program that processes ~ 25,000 jobs and each iteration
> > draws a histogram and writes out some of the output.  I let it run all
> > night and when I came back, python was filling up all my memory
> > (2Gigs) and was thrashing on and off of swap.  I narrowed the problem
> > down to my calling of the hist() function, and was able to reproduce
> > it in the following code I copied from the mpl website.
> >
> > Am I not properly closing the figure somehow?
> > Has this issue already been addressed?
> >
> > I recently installed version 0.90.1 of matplotlib, although I don't
> > see any easy way to verify that version number from within python.
> >
> > -Luke Robison
> >
> >
> > Code:
> > -------------------
> > import os,time,sys
> > from pylab import *
> >
> > def report_memory(i):
> >     pid = os.getpid()
> >     a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
> >     print i, '  ', a2[1],
> >     return int(a2[1].split()[1])
> >
> > # take a memory snapshot on indStart and compare it with indEnd
> > indStart, indEnd = 100, 150
> > for i in range(indStart,indEnd):
> >     ind = arange(100)
> >     xx = rand(len(ind))
> >
> >     figure(1)
> >     hist(xx)
> >     close(1)
> >
> > # wait a few cycles for memory usage to stabilize
> >     if i==indStart: start = val
> >     if i>indStart:
> >         end = val
> >         print 'Average memory consumed per loop: %1.4fk bytes' % \
> >               ((end-start)/float(indEnd-indStart))
> >
> > -----------------
> >
> > Output:
> >
> >
> > python memtest.py
> >
> > Average memory consumed per loop: 0.0000k bytes
> > 102    39808 21991
> > Average memory consumed per loop: 0.0000k bytes
> > 103    39828 21991
> > Average memory consumed per loop: 0.0000k bytes
> > 104    39852 22024
> > Average memory consumed per loop: 0.6600k bytes
> > 105    39876 22024
> > Average memory consumed per loop: 0.6600k bytes
> > 106    39908 22024
> > Average memory consumed per loop: 0.6600k bytes
> > 107    39932 22024
> > Average memory consumed per loop: 0.6600k bytes
> > 108    39960 22024
> > Average memory consumed per loop: 0.6600k bytes
> > 109    39980 22057
> > Average memory consumed per loop: 1.3200k bytes
> > 110    40008 22057
> > Average memory consumed per loop: 1.3200k bytes
> > 111    40032 22057
> > Average memory consumed per loop: 1.3200k bytes
> > 112    40056 22057
> > Average memory consumed per loop: 1.3200k bytes
> > 113    40084 22057
> > Average memory consumed per loop: 1.3200k bytes
> > 114    40104 22090
> > Average memory consumed per loop: 1.9800k bytes
> > 115    40132 22090
> > Average memory consumed per loop: 1.9800k bytes
> > 116    40156 22090
> > Average memory consumed per loop: 1.9800k bytes
> > 117    40180 22090
> > Average memory consumed per loop: 1.9800k bytes
> > 118    40208 22090
> > Average memory consumed per loop: 1.9800k bytes
> > 119    40232 22123
> > Average memory consumed per loop: 2.6400k bytes
> > 120    40256 22123
> > Average memory consumed per loop: 2.6400k bytes
> > 121    40280 22123
> > Average memory consumed per loop: 2.6400k bytes
> > 122    40304 22123
> > Average memory consumed per loop: 2.6400k bytes
> > 123    40328 22123
> > Average memory consumed per loop: 2.6400k bytes
> > 124    40356 22123
> > Average memory consumed per loop: 2.6400k bytes
> > 125    40380 22156
> > Average memory consumed per loop: 3.3000k bytes
> > 126    40404 22156
> > Average memory consumed per loop: 3.3000k bytes
> > 127    40428 22156
> > Average memory consumed per loop: 3.3000k bytes
> > 128    40452 22156
> > Average memory consumed per loop: 3.3000k bytes
> > 129    40476 22156
> > Average memory consumed per loop: 3.3000k bytes
> > 130    40500 22189
> > Average memory consumed per loop: 3.9600k bytes
> > 131    40528 22189
> > Average memory consumed per loop: 3.9600k bytes
> > 132    40548 22189
> > Average memory consumed per loop: 3.9600k bytes
> > 133    40576 22189
> > Average memory consumed per loop: 3.9600k bytes
> > 134    40596 22189
> > Average memory consumed per loop: 3.9600k bytes
> > 135    40624 22222
> > Average memory consumed per loop: 4.6200k bytes
> > 136    40652 22222
> > Average memory consumed per loop: 4.6200k bytes
> > 137    40676 22222
> > Average memory consumed per loop: 4.6200k bytes
> > 138    40700 22222
> > Average memory consumed per loop: 4.6200k bytes
> > 139    40724 22222
> > Average memory consumed per loop: 4.6200k bytes
> > 140    40744 22222
> > Average memory consumed per loop: 4.6200k bytes
> > 141    40768 22255
> > Average memory consumed per loop: 5.2800k bytes
> > 142    40800 22255
> > Average memory consumed per loop: 5.2800k bytes
> > 143    40824 22255
> > Average memory consumed per loop: 5.2800k bytes
> > 144    40848 22255
> > Average memory consumed per loop: 5.2800k bytes
> > 145    40872 22255
> > Average memory consumed per loop: 5.2800k bytes
> > 146    40896 22288
> > Average memory consumed per loop: 5.9400k bytes
> > 147    40916 22288
> > Average memory consumed per loop: 5.9400k bytes
> > 148    40940 22288
> > Average memory consumed per loop: 5.9400k bytes
> > 149    40972 22288
> > Average memory consumed per loop: 5.9400k bytes
> >
> >
> > as you can see, the memory consumption is increasing each loop, and
> > furthermore, and an increasing rate :-(
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems?  Stop.
> > Now Search log events and configuration files using AJAX and a browser.
> > Download your FREE copy of Splunk now >>  http://get.splunk.com/
> > _______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>