matplotlib-devel

Re: [matplotlib-devel] Memory leaks

[Eric F. <ef...@ha...>]

Michael Droettboom wrote:
> Interesting...
> 
> When you get a chance, would you mind running the attached script?  This 
> is how I was finding object leaks before.  It takes a single commandline 
> argument that is the number of iterations.  Can you send me the outputs 
> from 1 and 2 iterations?  That way we should be able to see what type of 
> object is being leaked, which is a good first step.

And here is the result of the script modified for gtk:

efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 1
55352 55417
*** <type 'gtk.gdk.Colormap'>
*** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
*** <class 'gtk._gtk.WidgetFlags'>
*** <class 'gtk._gtk.WidgetFlags'>
*** <type 'gtk.gdk.Window'>
*** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'>
*** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'>
*** <type 'gtk.gdk.Pixmap'>
*** <type 'gtk.Tooltips'>
*** <type 'gtk.Label'>
*** <type 'gtk.Window'>
*** <type 'gtk.VBox'>

uncollectable list: []

efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 2
55352 55421
*** <type 'gtk.gdk.Colormap'>
*** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
*** <class 'gtk._gtk.WidgetFlags'>
*** <class 'gtk._gtk.WidgetFlags'>
*** <type 'gtk.gdk.Window'>
*** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
*** <class 'gtk._gtk.WidgetFlags'>
*** <class 'gtk._gtk.WidgetFlags'>
*** <type 'gtk.gdk.Window'>
*** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'>
*** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'>
*** <type 'gtk.gdk.Pixmap'>
*** <type 'gtk.Tooltips'>
*** <type 'gtk.Label'>
*** <type 'gtk.Window'>
*** <type 'gtk.VBox'>

uncollectable list: []

Eric

Re: [matplotlib-devel] Memory leaks

[Ken M. <mc...@ii...>]

On Jul 5, 2007, at 2:13 PM, Michael Droettboom wrote:
>
> Interesting.  I don't get that, but I do get some random segfaults (I
> got lucky the first time I tested).

It looks like wxPython doesn't retain a reference to the wxApp PyObj  
for you:

	kmcivor@400exp209:~/Projects/matplotlib-svn$ pythonw
	Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
	[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
	Type "help", "copyright", "credits" or "license" for more information.
	>>> import wx
	>>> app = wx.PySimpleApp()
	>>> del app
	>>> wx.GetApp()
	Segmentation fault

> I'm awfully surprised that wx.GetApp() would return an iterator, as  
> you
> are getting, so maybe it's corruption of some sort?

My guess is that Eric got lucky and ob_type was pointing to the  
listiterator's C type instance.

> Since I didn't want to just put the wxapp global variable back in,  
> I assigned it to the figure that creates it, therefore stick around  
> as long as the figure does.  (Is that the correct thing for its  
> lifetime?)

I don't think this will work if you create two figures, destroy the  
first one, and then create another figure.  Once created, the wxApp  
needs to exist for the life of the python process.  I'll go ahead an  
put the global variable back in.

> Also, I'm a little puzzled by this code in show() in backend_wx.py:
>
>         wxapp = wx.GetApp()
>         if wxapp is not None:
>             # wxPython 2.4 has no wx.App.IsMainLoopRunning() method
>             imlr = getattr(wxapp, 'IsMainLoopRunning', lambda: False)
>             if imlr():
>                 wxapp.MainLoop()
>
> If I'm reading this correctly, shouldn't it be "if not imlr()"?

Yes, it should be.  I'll try to code with my eyes open from now on.  :-/

Ken

Re: [matplotlib-devel] Memory leaks

[Eric F. <ef...@ha...>]

Ken McIvor wrote:
> On Jul 5, 2007, at 2:13 PM, Michael Droettboom wrote:
>>
>> Interesting.  I don't get that, but I do get some random segfaults (I
>> got lucky the first time I tested).
> 
> It looks like wxPython doesn't retain a reference to the wxApp PyObj for 
> you:
> 
>     kmcivor@400exp209:~/Projects/matplotlib-svn$ pythonw
>     Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
>     [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> import wx
>     >>> app = wx.PySimpleApp()
>     >>> del app
>     >>> wx.GetApp()
>     Segmentation fault

This qualifies as a wx bug, doesn't it?  If wx doesn't retain the 
reference, then instead of a segfault shouldn't it raise an exception?

Eric

Re: [matplotlib-devel] Memory leaks

[Ken M. <mc...@ii...>]

On Jul 5, 2007, at 3:48 PM, Eric Firing wrote:
>
> This qualifies as a wx bug, doesn't it?

I believe so.  I'll file it.

> If wx doesn't retain the reference, then instead of a segfault  
> shouldn't it raise an exception?

I'd expect wx.GetApp() to work like the rest of wxPython and always  
return the wx.App instance.

This has been fixed in revision 3463.

Ken

Re: [matplotlib-devel] Memory leaks

[Christopher B. <Chr...@no...>]

Ken McIvor wrote:
>> This qualifies as a wx bug, doesn't it?
> I believe so.  I'll file it.

I agree - a segfault is ALWAYS a bug.

>> If wx doesn't retain the reference, then instead of a segfault  
>> shouldn't it raise an exception?
> 
> I'd expect wx.GetApp() to work like the rest of wxPython and always  
> return the wx.App instance.

If a wx.App has not been created, it returns None:

 >>> import wx
 >>> wx.GetApp()
 >>> a = wx.GetApp()
 >>> print a
None

Which is probably what it should do if the wxApp() has been deleted.

In any case, you can only create one wxApp per program instance, and it 
can not be destroyed and re-started, so keeping a global instance around 
is probably the way to go.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [matplotlib-devel] Memory leaks

[Michael D. <md...@st...>]

More results:

I've built and tested a more recent pygtk+ stack.  (glib-2.12, 
gtk+-2.10.9, librsvg-2.16.1, libxml2-2.6.29, pygobject-2.13.1, 
pygtk-2.10.4...).  The good news is that the C-level leaks I was seeing 
in pygtk 2.2 and 2.4 are resolved.  In particular, using an SVG icon and 
Gdk rendering no longer seems problematic.  I would suggest that anyone 
using old versions of pygtk should upgrade, rather than spending time on 
workarounds for matplotlib -- do you all agree?  And my Gtk patch should 
probably be reverted to use an SVG icon for the window again (or to only 
do it on versions of pygtk > 2.xx).  I don't know what percentage of 
users are still using pygtk-2.4 and earlier...

There is, however, a new patch (attached) to fix a leak of 
FileChooserDialog objects that I didn't see in earlier pygtk versions.  
I have to admit that I'm a bit puzzled by the solution -- it seems that 
the FileChooserDialog object refuses to destruct whenever any custom 
Python attributes have been added to the object.  It doesn't really need 
them in this case so it's an easy fix, but I'm not sure why that was 
broken -- other classes do this and don't have problems (e.g. 
NavigationToolbar2GTK).  Maybe a pygtk expert out there knows what this 
is about.  It would be great if this resolved the linear memory growth 
that Eric is seeing with the Gtk backend.

GtkCairo seems to be free of leaks.

QtAgg (qt-3.3) was leaking because of a cyclical reference in the 
signals between the toolbar and its buttons.  (Patch attached).

Qt4 is forthcoming (I'm still trying to compile something that runs the 
demos cleanly). 

I tried the FltkAgg backend, but it doesn't seem to close the window at 
all when the figure is closed -- instead I get dozens of windows open at 
once.  Is that a known bug or correct behavior?

Cheers,
Mike

Eric Firing wrote:
> Michael Droettboom wrote:
>> Eric Firing wrote:
>>> So, this test is still showing problems, with similar memory 
>>> consumption in these three backends.
>> Not necessarily.  By default, Python allocates large pools from the 
>> operating system and then manages those pools itself (though its 
>> PyMalloc call).  Prior to Python 2.5, those pools were never freed.  
>> With Python 2.5, empty pools, when they occur, are freed back to the 
>> OS.  Due to fragmentation issues, even if there is enough free space 
>> in those pools for new objects, new pools may need to be created 
>> anyway, since Python objects can't be moved once they are created.  
>> So seeing modest increases in memory usage during a long-running 
>> Python application is typical, and not something that can be avoided 
>> wiinaccurate at finding memory leaksthout micro-optimizing for pool 
>> performance (something that may be very difficult).  If memory usage 
>> is truly increasing in an unbounded way, then, yes, there may be 
>> problems, but it should eventually stabilize (though in a test such 
>> as memleak_gui that may take many iterations).  It's more interesting 
>> to see the curve of memory usage over time than the average over a 
>> number of iterations.
>
> I agree.  I just ran 2000 iterations with GtkAgg, plotted every 10th 
> point, and the increase is linear (apart from a little bumpiness) over 
> the entire range (not just the last 1000 iterations reported below):
>
> Backend GTKAgg, toolbar toolbar2
> Averaging over loops 1000 to 2000
> Memory went from 31248k to 35040k
> Average memory consumed per loop: 3.7920k bytes
>
> Maybe this is just the behavior of pymalloc in 2.5?
>
>
>> For further reading, see:
>> http://evanjones.ca/python-memory.html
>> README.valgrind in the Python source
>> http://mail.python.org/pipermail/python-dev/2006-March/061991.html
>>
>> Because of this, using the total memory allocated by the Python 
>> process to track memory leaks is pretty blunt tool.  More important 
>> metrics are the total number of GC objects (gc.get_objects()), GC 
>> garbage (gc.garbage), and using a tool like Valgrind or Purify to 
>> find mismatched malloc/frees.  Another useful tool (but I didn't 
>> resort to yet with matplotlib testing) is to build Python with 
>> COUNT_ALLOCS, which then gives access to the total number of mallocs 
>> and frees in the Python interpreter at runtime.
>>
>> IMO, the only reasonable way to use the total memory usage of Python 
>> to debug memory leaks is if you build Python without pool allocation 
>> (--without-pymalloc).  That was how I was debugging memory leaks last 
>> week (in conjunction with valgrind, and the gc module), and with that 
>> configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
>> very small amount with Tk.  Are your numbers from a default build?  
>> If so, I'll rebuild my Python and check my numbers against yours.  If 
>> they match, I suspect there's little we can do.
>
> I used stock Python 2.5 from ubuntu Feisty.  I should compile a 
> version as you suggest, but I haven't done it yet.
>
> Eric
>
>>
>> Cheers,
>> Mike
>>
>
>

Re: [matplotlib-devel] Memory leaks

[Michael D. <md...@st...>]

Eric Firing wrote:
>
> I just committed a change to the output formatting of memleak_gui so 
> that if you redirect it to a file, that file can be loaded with 
> pylab.load() in case you want to plot the columns.  (At least this is 
> true if you don't use the -c option.)
>
Great.  Sorry for stomping on that ;)
> Yesterday, before your commits, I compared memleak_gui with stock 
> Python 2.4 versus stock 2.5 (both from ubuntu feisty) and found very 
> little difference in the OS memory numbers.
Are they still increasing linearly?  I'm still seeing some mystery leaks 
with Gtk, Qt4 and (much smaller) on Tk.  Qt and Wx seem fine here.  
Unfortunately Qt4 crashes valgrind, so it's not of much use. 

I'm curious whether your results match that.  I'm not terribly surprised 
that 2.4 isn't different from 2.5, since the case in which entire memory 
pools are freed in 2.5 is probably hard to trigger.

Cheers,
Mike

Re: [matplotlib-devel] Memory leaks

[Michael D. <md...@st...>]

Eric Firing wrote:
> Attached are runs with gtk, wx, qtagg, and tkagg.  Quite a variety of 
> results: tkagg is best, with only slow memory growth and a constant 
> number of python objects; qtagg grows by 2.2k per loop, with no 
> increase in python object count; wx (which is built on gtk) consumes 
> 3.5k per loop, with an increasing object count; gtk consumes 1.8k per 
> loop with an increasing object count.
>
> All runs are on stock ubuntu feisty python 2.5.
Thanks for these results.  Unfortunately, I'm seeing different results 
here.  [dagnabbit!]  None of them have an increasing object count for 
me, which leads me to suspect there's some version difference between 
your environment and mine that isn't being accounted for.

Gtk[Agg|Cairo] -- 1.3k per loop.
Wx[Agg] -- 0.010k per loop
QtAgg -- 2.3k per loop (which is in the same ballpark as your result)
Qt4Agg -- 1.4k per loop (which seems to be in the same ballpark as 
Darren Dale's result)
TkAgg -- 0.29k per loop

I don't know if the size of memory per loop is directly comparable 
between your environment and mine, but certainly the shape of the curve, 
and whether the number of Python objects is growing is very relevant.

I made some more commits to SVN on 07/03/07 necessary for recent 
versions of gtk+ and qt.  Did you (by any chance) not get those 
patches?  It would also be interesting to know which versions of the 
toolkits you have, as they are probably different from mine.  Is it safe 
to assume that they are all the stock Ubuntu feisty packages?  In any 
case, I have updated memleak_gui.py to display the relevant toolkit 
versions.  I've also attached a script to display the toolkit versions.  
Its output on my machine is:

# pygtk version: (2, 10, 4), gtk version: (2, 10, 9)
# PyQt4 version: 4.2, Qt version 40300
# pyqt version: 3.17.2, qt version: 30303
# wxPython version: 2.8.4.0
# Tkinter version: $Revision: 50704 $, Tk version: 8.4, Tcl version: 8.4

Cheers,
Mike

Re: [matplotlib-devel] Memory leaks

[Michael D. <md...@st...>]

Eric,

It turns out that the key difference is in pygobject.  I was using 
2.13.2 (from source) here at work, and Ubuntu Feisty currently ships 
2.12.3.  2.13.x is an unstable branch, of course, (gnome-related 
packages follow the even/odd versioning scheme), so I shouldn't have 
been using it to test with in the first place...

...however, there are only about 40 patches between 2.12.3 and 2.13.2.  
Two of them are very crucial memory leak patches:

    
http://svn.gnome.org/viewcvs/pygobject?limit_changes=100&view=revision&revision=653
    
http://svn.gnome.org/viewcvs/pygobject?limit_changes=100&view=revision&revision=642

When I patched 2.12.3 against these patches, the Python object leaks go 
away.  There may be workarounds matplotlib can make -- I'll look into 
that now.  Hopefully the workarounds won't be too problematic -- I would 
hate to complicate matplotlib code for the sake of something that should 
resolve itself in the next Gnome release cycle...

I updated the memleak_gui.py script to display the pygobject/glib 
versions as well when testing the gtk backend.

Cheers,
Mike

Michael Droettboom wrote:
> Eric Firing wrote:
>   
>> Michael Droettboom wrote:
>>     
>>> Interesting...
>>>
>>> When you get a chance, would you mind running the attached script?  
>>> This is how I was finding object leaks before.  It takes a single 
>>> commandline argument that is the number of iterations.  Can you send 
>>> me the outputs from 1 and 2 iterations?  That way we should be able 
>>> to see what type of object is being leaked, which is a good first step.
>>>       
>> And here is the result of the script modified for gtk:
>>
>> efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 1
>> 55352 55417
>> *** <type 'gtk.gdk.Colormap'>
>> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
>> *** <class 'gtk._gtk.WidgetFlags'>
>> *** <class 'gtk._gtk.WidgetFlags'>
>> *** <type 'gtk.gdk.Window'>
>> *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'>
>> *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'>
>> *** <type 'gtk.gdk.Pixmap'>
>> *** <type 'gtk.Tooltips'>
>> *** <type 'gtk.Label'>
>> *** <type 'gtk.Window'>
>> *** <type 'gtk.VBox'>
>>
>> uncollectable list: []
>>
>> efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 2
>> 55352 55421
>> *** <type 'gtk.gdk.Colormap'>
>> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
>> *** <class 'gtk._gtk.WidgetFlags'>
>> *** <class 'gtk._gtk.WidgetFlags'>
>> *** <type 'gtk.gdk.Window'>
>> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
>> *** <class 'gtk._gtk.WidgetFlags'>
>> *** <class 'gtk._gtk.WidgetFlags'>
>> *** <type 'gtk.gdk.Window'>
>> *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'>
>> *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'>
>> *** <type 'gtk.gdk.Pixmap'>
>> *** <type 'gtk.Tooltips'>
>> *** <type 'gtk.Label'>
>> *** <type 'gtk.Window'>
>> *** <type 'gtk.VBox'>
>>
>> uncollectable list: []
>>
>> Eric
>>
>>     
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
>   

Re: [matplotlib-devel] Memory leaks

[John H. <jd...@gm...>]

On 7/9/07, Michael Droettboom <md...@st...> wrote:

> ...however, there are only about 40 patches between 2.12.3 and 2.13.2.
> Two of them are very crucial memory leak patches:

> When I patched 2.12.3 against these patches, the Python object leaks go
> away.  There may be workarounds matplotlib can make -- I'll look into
> that now.  Hopefully the workarounds won't be too problematic -- I would
> hate to complicate matplotlib code for the sake of something that should
> resolve itself in the next Gnome release cycle...

I wouldn't spend any time on working around a leak that is already
fixed in gobject.  We just need to update the FAQ so GTK users will
know that if they have a problem with leaks in GTK*, they need to
upgrade.

JDH

Re: [matplotlib-devel] Memory leaks

[Christopher B. <Chr...@no...>]

Eric Firing wrote:
> I just updated from svn and tried to rerun the wx test, but ran into an 
> error:
> 
> efiring@manini:~/programs/py/mpl/tests$ python 
>      wxapp.Yield()
> NameError: global name 'wxapp' is not defined

I think I just saw a note that Ken had committed a patch that a user had 
provided that kept the wx back-end from re-starting an event loop if 
there was one already running -- maybe that has something to do with 
this bug?

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [matplotlib-devel] Memory leaks

[Michael D. <md...@st...>]

Yes -- the global wxapp variable was removed (a very good thing).  I 
just committed a patch to fix this crash (r3460)

Cheers,
Mike

Christopher Barker wrote:
> Eric Firing wrote:
>   
>> I just updated from svn and tried to rerun the wx test, but ran into an 
>> error:
>>
>> efiring@manini:~/programs/py/mpl/tests$ python 
>>      wxapp.Yield()
>> NameError: global name 'wxapp' is not defined
>>     
>
> I think I just saw a note that Ken had committed a patch that a user had 
> provided that kept the wx back-end from re-starting an event loop if 
> there was one already running -- maybe that has something to do with 
> this bug?
>
> -Chris
>
>
>
>
>   

Re: [matplotlib-devel] Memory leaks

[Eric F. <ef...@ha...>]

Mike,

New exception:

efiring@manini:~/programs/py/mpl/tests$ python 
../matplotlib_units/unit/memleak_gui.py -dwx -s1000 -e2000 > 
~/temp/memleak_wx_0705.asc
Traceback (most recent call last):
   File "../matplotlib_units/unit/memleak_gui.py", line 58, in <module>
     pylab.close(fig)
   File "/usr/local/lib/python2.5/site-packages/matplotlib/pylab.py", 
line 742, in close
     _pylab_helpers.Gcf.destroy(manager.num)
   File 
"/usr/local/lib/python2.5/site-packages/matplotlib/_pylab_helpers.py", 
line 28, in destroy
     figManager.destroy()
   File 
"/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py", 
line 1405, in destroy
     self.frame.Destroy()
   File 
"/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py", 
line 1364, in Destroy
     wxapp.Yield()
AttributeError: 'listiterator' object has no attribute 'Yield'


Eric

Re: [matplotlib-devel] Memory leaks

[Michael D. <md...@st...>]

Interesting.  I don't get that, but I do get some random segfaults (I 
got lucky the first time I tested).

I'm awfully surprised that wx.GetApp() would return an iterator, as you 
are getting, so maybe it's corruption of some sort?

Reverting to revision 3441 on backend_wx.py does resolve this issue for 
me, so it is related to removing the wxapp global variable.  While I 
like the idea of removing global variables, that was problematic, since 
when the wxapp variable is dereferenced, the whole wx.App is destructed, 
(hence, I believe the segfaults).  Since I didn't want to just put the 
wxapp global variable back in, I assigned it to the figure that creates 
it, therefore stick around as long as the figure does.  (Is that the 
correct thing for its lifetime?)  Anyway, it seems to fix memleak_gui.py 
for me.  Ken and Tim will probably want to check that I didn't cause 
more mainloops to start than necessary in the process.

Also, I'm a little puzzled by this code in show() in backend_wx.py:

        wxapp = wx.GetApp()
        if wxapp is not None:
            # wxPython 2.4 has no wx.App.IsMainLoopRunning() method
            imlr = getattr(wxapp, 'IsMainLoopRunning', lambda: False)
            if imlr():
                wxapp.MainLoop()

If I'm reading this correctly, shouldn't it be "if not imlr()"?  If it 
is correct, maybe it needs a comment as to why mainloops should be 
started if a mainloop is already running.

Cheers,
Mike

Eric Firing wrote:
> Mike,
>
> New exception:
>
> efiring@manini:~/programs/py/mpl/tests$ python 
> ../matplotlib_units/unit/memleak_gui.py -dwx -s1000 -e2000 > 
> ~/temp/memleak_wx_0705.asc
> Traceback (most recent call last):
>   File "../matplotlib_units/unit/memleak_gui.py", line 58, in <module>
>     pylab.close(fig)
>   File "/usr/local/lib/python2.5/site-packages/matplotlib/pylab.py", 
> line 742, in close
>     _pylab_helpers.Gcf.destroy(manager.num)
>   File 
> "/usr/local/lib/python2.5/site-packages/matplotlib/_pylab_helpers.py", 
> line 28, in destroy
>     figManager.destroy()
>   File 
> "/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py", 
> line 1405, in destroy
>     self.frame.Destroy()
>   File 
> "/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py", 
> line 1364, in Destroy
>     wxapp.Yield()
> AttributeError: 'listiterator' object has no attribute 'Yield'
>
>
> Eric
>

[matplotlib-devel] Memory leaks FAQ (was: Re: Memory leaks)

[Michael D. <md...@st...>]

I am about to update the memory leak question in the FAQ, but I thought 
I'd run it by the list first.  I removed language that talked about much 
earlier releases of mpl, and the paragraph about leaks in older versions 
of Numeric and numarray.  It seems like we should recommend numpy when 
the user experiences problems with Numeric or numarray, but I wanted to 
confirm that before adding a paragraph to that effect.

The second question (below), can be ommitted -- it really belongs in a 
developer's FAQ.  Any recommendations on a better place to post that 
information?

<snip>


      matplotlib appears to be leaking memory, what should I do?

First, determine if it is a true memory leak. Python allocates memory in 
pools, rather than one object at a time, which can make memory leaks 
difficult to diagnose. Memory usage may appear to increase rapidly with 
each iteration, but should eventually reach a steady state if no true 
memory leaks exist. (For more fine-grained memory usage reporting, you 
can build a custom Python with the --without-pymalloc flag, which 
disables pool allocation.) If after sufficient iterations, you still see 
that memory usage is increasing, it is a likely a bonafide memory leak 
that should be reported.

memleak_gui.py 
<http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/unit/memleak_gui.py?view=markup> 
(in the unit directory of the source tree), contains an example script 
useful for diagnosing and reporting memory leaks. Please use something 
like it when reporting leaks so we get an idea of the magnitude of the 
problem (i.e. bytes per figure). Also please provide your platform, 
matplotlib version, backend and as much information about the versions 
of the libraries you are using: freetype, png and zlib. It would also 
help if you posted code so I could look for any obvious coding errors 
vis-a-vis matplotlib.

There are some known memory leaks in matplotlib-0.90.1 when used in 
conjunction with the Tk, Gtk, Wx, Qt and Qt4 GUI backends. Many of these 
leaks have resolutions in the current SVN version of matplotlib. 
However, the following library versions are known to have leaks that 
matplotlib triggers. If you have one of these versions and are 
experiencing memory leaks, you should upgrade your library. This is not 
an exhaustive list.

    * *Wx backend:*wxPython-2.8.2 or earlier in the 2.8 series
    * *Gtk backend:*pygobject-2.12.x, pygtk-2.4.0


      I'd like to help diagnose a memory leak, rather than just report
      it. How do you recommend I do that?

I thought you'd never ask! Python memory leaks tend to fall into one of 
the following categories:

    * *Uncollectable garbage:* The Python garbage collector is not
      infallible. It can not collect objects that the Python that have
      __del__ methods or weakrefs and contain cycles. If curious, you
      can read about this problem in horrific detail
      <http://svn.python.org/view/python/trunk/Modules/gc_weakref.txt?view=markup>.
      You can obtain a list of all uncollectable objects as follows:

                      import gc
                      print gc.garbage
          

      To see what cycles these objects participate in, there is a useful
      function in matplotlib.cbook:

                      from matplotlib import cbook
      		cbook.print_cycles(gc.garbage)
          

      This will print out all of the reference cycles that are
      preventing the uncollectable objects from being freed. The code
      should then be modified to prevent these cycles, or break the
      cycles during destruction.
    * *Real references:* Sometimes objects are legitimately being held
      onto by other Python objects. When this happens, you would see the
      total number of Python objects in the interpreter increase with
      each iteration of your test, even when you didn't intend any data
      to "stick around". You can print out the total number of objects
      in the interpreter:

                      import gc
      		print len(gc.get_objects())
          

      By comparing the objects before and after your test, you can
      determine which of them remain between iterations:

                      original_objects = [id(x) for x in gc.get_objects()]
      		# ... do something that leaks objects
      		new_objects = [x for x in gc.get_objects() 
      		               if id(x) not in original_objects]
          

      You can then determine what is referencing those objects and
      causing them to stick around:

                      print gc.get_referents(x)
          

      The code should be modified to prevent these unwanted references.
    * *C/C++ leaks in extension objects:* These is the classic problem
      of objects that are "malloc'd/new'd" and never "freed/deleted" in
      C or C++ parlance. There are many memory debuggers available such
      as Purify or Valgrind to help find these errors. I recommend
      reading the documentation about using these tools in conjunction
      with Python
      <http://svn.python.org/view/python/trunk/Misc/README.valgrind?view=markup>
      to avoid a lot of false positives.

Re: [matplotlib-devel] Memory leaks FAQ (was: Re: Memory leaks)

[John H. <jd...@gm...>]

On 7/9/07, Michael Droettboom <md...@st...> wrote:
>
>  I am about to update the memory leak question in the FAQ, but I thought I'd
> run it by the list first.  I removed language that talked about much earlier
> releases of mpl, and the paragraph about leaks in older versions of Numeric
> and numarray.  It seems like we should recommend numpy when the user
> experiences problems with Numeric or numarray, but I wanted to confirm that
> before adding a paragraph to that effect.

vis-a-vis numpy, since we are dropping support entirely for Numeric
and numarray, you can definitely recommend numoy :-)

The looks very good, but I would also point them to memleak_hawaii or
something to diagnose a pure image backend.  Many times it is simply
the user script that is failing (they fail to properly close a figure
they have opened) so the FAQ should at least point them to an image
generation script that is written correctly.  The GUI leaks are are
current problem, but they are 2nd order.  We first need to make sure
the pure MPL part is not leaking (it isn't currently, but it might one
day if we screw something up in extension code) and once we've
verified that, we can isolate the GUI leaks.

Thanks,
JDH

Re: [matplotlib-devel] Memory leaks

[Michael D. <md...@st...>]

That is at least something to go by.  ;)

Thanks,
Mike

Eric Firing wrote:
> Michael Droettboom wrote:
>> Interesting...
>>
>> When you get a chance, would you mind running the attached script?  
>> This is how I was finding object leaks before.  It takes a single 
>> commandline argument that is the number of iterations.  Can you send 
>> me the outputs from 1 and 2 iterations?  That way we should be able 
>> to see what type of object is being leaked, which is a good first step.
>
> And here is the result of the script modified for gtk:
>
> efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 1
> 55352 55417
> *** <type 'gtk.gdk.Colormap'>
> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
> *** <class 'gtk._gtk.WidgetFlags'>
> *** <class 'gtk._gtk.WidgetFlags'>
> *** <type 'gtk.gdk.Window'>
> *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'>
> *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'>
> *** <type 'gtk.gdk.Pixmap'>
> *** <type 'gtk.Tooltips'>
> *** <type 'gtk.Label'>
> *** <type 'gtk.Window'>
> *** <type 'gtk.VBox'>
>
> uncollectable list: []
>
> efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 2
> 55352 55421
> *** <type 'gtk.gdk.Colormap'>
> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
> *** <class 'gtk._gtk.WidgetFlags'>
> *** <class 'gtk._gtk.WidgetFlags'>
> *** <type 'gtk.gdk.Window'>
> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'>
> *** <class 'gtk._gtk.WidgetFlags'>
> *** <class 'gtk._gtk.WidgetFlags'>
> *** <type 'gtk.gdk.Window'>
> *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'>
> *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'>
> *** <type 'gtk.gdk.Pixmap'>
> *** <type 'gtk.Tooltips'>
> *** <type 'gtk.Label'>
> *** <type 'gtk.Window'>
> *** <type 'gtk.VBox'>
>
> uncollectable list: []
>
> Eric
>

[matplotlib-devel] wxPython reference leak was (Re: Memory leaks)

[Michael D. <md...@st...>]

I had no trouble reproducing this on my Ubuntu Feisty box.

It turns out that wxPython leaks a dictionary for every object whose
class subclasses a Wx class.  There is a fix for this that made it into
wxPython-2.8.3.0:

http://cvs.wxwidgets.org/viewcvs.cgi/wxWidgets/wxPython/src/helpers.cpp.diff?r1=1.145&r2=1.145.4.1 


I have verified this on my source-built wxPython-2.8.4.0.  If I remove
this line, I can reproduce the reference leak.

** I would recommend that anyone using a wxPython-2.8.x prior to
wxPython-2.8.4 upgrade.  There are binary packages available for
a number of distributions on wxpython.org. **

As an aside, I filed a bug for this on Ubuntu launchpad.  I don't know
if this qualifies for the kind of fix they would normally make as a
maintenance release, but I thought it was worth trying.

https://bugs.launchpad.net/ubuntu/+source/wxwidgets2.8/+bug/124381

Cheers,
Mike

Eric Firing wrote:
> Michael Droettboom wrote:
>> Interesting...
>>
>> When you get a chance, would you mind running the attached script?  
>> This is how I was finding object leaks before.  It takes a single 
>> commandline argument that is the number of iterations.  Can you send 
>> me the outputs from 1 and 2 iterations?  That way we should be able 
>> to see what type of object is being leaked, which is a good first step.
>
> efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_wx.py 1
> 75891 76010
> *** <class 'wx._core.PySimpleApp'>
> *** <class 'wx._core._wxPyDeadObject'>
>
> uncollectable list: []
>
> efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_wx.py 2
> GnomePrintCupsPlugin-Message: The ppd file for the CUPS printer 
> Dell424 could not be loaded.
> GnomePrintCupsPlugin-Message: The ppd file for the CUPS printer pslj4m 
> could not be loaded.
> 75891 76014
> *** <class 'wx._core.PySimpleApp'>
> *** <class 'wx._core._wxPyDeadObject'>
>
> uncollectable list: []
>
>
> Eric
>

[matplotlib-devel] E-mail server trouble

[Michael D. <md...@st...>]

My sincere apologies for the multiple copies of the e-mail sent this 
morning.  I was getting "SMTP server down" messages, but clearly the 
messages were sent anyway.

I'm not a spammer, really! ;)

Mike