Re: [matplotlib-devel] Memory leaks

[Michael D. <md...@st...>]

Forgot to attach the patches.

Oops,
Mike

Michael Droettboom wrote:
> More results:
>
> I've built and tested a more recent pygtk+ stack.  (glib-2.12, 
> gtk+-2.10.9, librsvg-2.16.1, libxml2-2.6.29, pygobject-2.13.1, 
> pygtk-2.10.4...).  The good news is that the C-level leaks I was seeing 
> in pygtk 2.2 and 2.4 are resolved.  In particular, using an SVG icon and 
> Gdk rendering no longer seems problematic.  I would suggest that anyone 
> using old versions of pygtk should upgrade, rather than spending time on 
> workarounds for matplotlib -- do you all agree?  And my Gtk patch should 
> probably be reverted to use an SVG icon for the window again (or to only 
> do it on versions of pygtk > 2.xx).  I don't know what percentage of 
> users are still using pygtk-2.4 and earlier...
>
> There is, however, a new patch (attached) to fix a leak of 
> FileChooserDialog objects that I didn't see in earlier pygtk versions.  
> I have to admit that I'm a bit puzzled by the solution -- it seems that 
> the FileChooserDialog object refuses to destruct whenever any custom 
> Python attributes have been added to the object.  It doesn't really need 
> them in this case so it's an easy fix, but I'm not sure why that was 
> broken -- other classes do this and don't have problems (e.g. 
> NavigationToolbar2GTK).  Maybe a pygtk expert out there knows what this 
> is about.  It would be great if this resolved the linear memory growth 
> that Eric is seeing with the Gtk backend.
>
> GtkCairo seems to be free of leaks.
>
> QtAgg (qt-3.3) was leaking because of a cyclical reference in the 
> signals between the toolbar and its buttons.  (Patch attached).
>
> Qt4 is forthcoming (I'm still trying to compile something that runs the 
> demos cleanly). 
>
> I tried the FltkAgg backend, but it doesn't seem to close the window at 
> all when the figure is closed -- instead I get dozens of windows open at 
> once.  Is that a known bug or correct behavior?
>
> Cheers,
> Mike
>
> Eric Firing wrote:
>   
>> Michael Droettboom wrote:
>>     
>>> Eric Firing wrote:
>>>       
>>>> So, this test is still showing problems, with similar memory 
>>>> consumption in these three backends.
>>>>         
>>> Not necessarily.  By default, Python allocates large pools from the 
>>> operating system and then manages those pools itself (though its 
>>> PyMalloc call).  Prior to Python 2.5, those pools were never freed.  
>>> With Python 2.5, empty pools, when they occur, are freed back to the 
>>> OS.  Due to fragmentation issues, even if there is enough free space 
>>> in those pools for new objects, new pools may need to be created 
>>> anyway, since Python objects can't be moved once they are created.  
>>> So seeing modest increases in memory usage during a long-running 
>>> Python application is typical, and not something that can be avoided 
>>> wiinaccurate at finding memory leaksthout micro-optimizing for pool 
>>> performance (something that may be very difficult).  If memory usage 
>>> is truly increasing in an unbounded way, then, yes, there may be 
>>> problems, but it should eventually stabilize (though in a test such 
>>> as memleak_gui that may take many iterations).  It's more interesting 
>>> to see the curve of memory usage over time than the average over a 
>>> number of iterations.
>>>       
>> I agree.  I just ran 2000 iterations with GtkAgg, plotted every 10th 
>> point, and the increase is linear (apart from a little bumpiness) over 
>> the entire range (not just the last 1000 iterations reported below):
>>
>> Backend GTKAgg, toolbar toolbar2
>> Averaging over loops 1000 to 2000
>> Memory went from 31248k to 35040k
>> Average memory consumed per loop: 3.7920k bytes
>>
>> Maybe this is just the behavior of pymalloc in 2.5?
>>
>>
>>     
>>> For further reading, see:
>>> http://evanjones.ca/python-memory.html
>>> README.valgrind in the Python source
>>> http://mail.python.org/pipermail/python-dev/2006-March/061991.html
>>>
>>> Because of this, using the total memory allocated by the Python 
>>> process to track memory leaks is pretty blunt tool.  More important 
>>> metrics are the total number of GC objects (gc.get_objects()), GC 
>>> garbage (gc.garbage), and using a tool like Valgrind or Purify to 
>>> find mismatched malloc/frees.  Another useful tool (but I didn't 
>>> resort to yet with matplotlib testing) is to build Python with 
>>> COUNT_ALLOCS, which then gives access to the total number of mallocs 
>>> and frees in the Python interpreter at runtime.
>>>
>>> IMO, the only reasonable way to use the total memory usage of Python 
>>> to debug memory leaks is if you build Python without pool allocation 
>>> (--without-pymalloc).  That was how I was debugging memory leaks last 
>>> week (in conjunction with valgrind, and the gc module), and with that 
>>> configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
>>> very small amount with Tk.  Are your numbers from a default build?  
>>> If so, I'll rebuild my Python and check my numbers against yours.  If 
>>> they match, I suspect there's little we can do.
>>>       
>> I used stock Python 2.5 from ubuntu Feisty.  I should compile a 
>> version as you suggest, but I haven't done it yet.
>>
>> Eric
>>
>>     
>>> Cheers,
>>> Mike
>>>
>>>       
>>     
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
>