Re: [matplotlib-devel] Memory leaks

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Mike,

All this sounds like great progress--thanks!  I particularly appreciate 
the descriptions of what problems you found and how you found them.

John et al.: is there a maintainer for each of these backends?  I think 
it is very important that Mike's patches be checked out and applied ASAP 
if they are OK; or if there is a problem, then that info needs to get 
back to Mike.  This should be very high priority.  I can do a quick 
check and commit if necessary, but it would make more sense for someone 
more familiar with backends and gui code to do it.  Or at least for 
others to do some testing on different platforms if I make the commits.

Eric

Michael Droettboom wrote:
> I've been looking into the memory leaks exercised by the memleak_gui.py 
> unit test.  I've searched through the mailing list for information, but 
> I'm new to the party here, so forgive me if I'm not fully current.
> 
> Eric Firing wrote:
> "I think we have a similar problem with all interactive backends (the 
> only one I didn't test is Qt4Agg) which also makes me suspect we are 
> violating some gui rule, rather than that gtk, qt3, wx, and tk all have 
> leaks."
> 
> Unfortunately, from what I've seen, there isn't a single rule being 
> broken, but instead I've been running into lots of different "surprises" 
> in the various toolkits.  But I am starting to suspect that my old 
> versions of Gtk (or PyGtk) have some bonafide leaks.
> 
> I just finished submitting patches (to the tracker) for a number of 
> memory leaks in the Tk, Gtk, and Wx backends (other backends will 
> hopefully follow).  I did all my testing on RHEL4 with Python 2.5 and 
> recent SVN matplotlib (rev. 3244), so it's quite possible that memory 
> leaks still remain on other platforms.
> 
> Tk:
> See the patch:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1745400&group_id=80706&atid=560722 
> 
> 
> Even after this patch, Tkinter still leaks 28 bytes every time it is 
> initialized (every time a toplevel window is created).  There may be a 
> way to avoid the subsequent initializations, but it wasn't immediately 
> obvious to me, and given the small size of this leak, I've passed on it 
> for now.
> 
> Gtk:
> See the patch:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1745406&group_id=80706&atid=560722 
> 
> 
> The patch fixes a number of Python-level leaks.
> Unfortunately, the Gdk rendering backend still leaks gtk.gdk.Window 
> objects, and I have so far been unable to determine a fix.  GtkAgg, 
> however, does not have this leak.
> Under pygtk-2.4.0, the toolbars leak gdk.Pixbuf's and file selector 
> dialogs.
> Since these issues appear to be bugs in gtk and/or pygtk itself, I will 
> probably first try a more recent version of them.  (The gtk installed on 
> RHEL4 is ancient (2.4), and I haven't yet tried building my own.  If 
> anyone has a more recent version of pygtk handy, I would appreciate a 
> report from memleak_gui.py after applying my patch.)
> 
> Wx:
> See the patch:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1745408&group_id=80706&atid=560722 
> 
> 
> This one was fairly simple, though surprising.  Top-level windows do not 
> fully destroy themselves as long as there are pending events.  Manually 
> flushing the event queue causes the windows to go away.  See the 
> wxPython docs:
> http://wxwidgets.org/manuals/stable/wx_wxwindow.html#wxwindowdestroy
> 
> There were no further leaks in Wx/WxAgg that I was able to detect (in my 
> environment).
> 
> 
> As an aside, I thought I'd share the techniques I used to find these 
> leaks (hopefully not to be pedantic, but these things were hard to come 
> by online...), and it might be useful to some.
> 
> For C/C++ memory leaks, I really like valgrind (though it is 
> Linux-only), though be sure to follow the directions to get it to play 
> well with Python.  I recommend rebuilding Python with 
> "--without-pymalloc" to make memory reporting in general much more 
> sensical (though slower).  See:
>    http://svn.python.org/view/python/trunk/Misc/README.valgrind
> For an example, you can see the rsvg memory leak here:
> 
> ==15979== 1,280 bytes in 20 blocks are definitely lost in loss record 
> 13,506 of 13,885
> ==15979==    at 0x4004405: malloc (vg_replace_malloc.c:149)
> ==15979==    by 0x314941: (within /usr/lib/libart_lgpl_2.so.2.3.16)
> ==15979==    by 0x315E0C: (within /usr/lib/libart_lgpl_2.so.2.3.16)
> ==15979==    by 0x31624A: art_svp_intersector (in 
> /usr/lib/libart_lgpl_2.so.2.3.16)
> ==15979==    by 0x316660: art_svp_intersect (in 
> /usr/lib/libart_lgpl_2.so.2.3.16)
> ==15979==    by 0x6BFA86C: (within /usr/lib/librsvg-2.so.2.8.1)
> ==15979==    by 0x6BFD801: rsvg_render_path (in 
> /usr/lib/librsvg-2.so.2.8.1)
> ==15979==    by 0x6BFD9E7: rsvg_handle_path (in 
> /usr/lib/librsvg-2.so.2.8.1)
> ==15979==    by 0x6BFFDB5: rsvg_start_path (in /usr/lib/librsvg-2.so.2.8.1)
> ==15979==    by 0x6C07F59: (within /usr/lib/librsvg-2.so.2.8.1)
> ==15979==    by 0x9C24BB: xmlParseStartTag (in /usr/lib/libxml2.so.2.6.16)
> ==15979==    by 0xA4DADC: xmlParseChunk (in /usr/lib/libxml2.so.2.6.16)
> ==15979==    by 0x6C08ED5: rsvg_handle_write_impl (in 
> /usr/lib/librsvg-2.so.2.8.1)
> ==15979==    by 0x6C09539: rsvg_handle_write (in 
> /usr/lib/librsvg-2.so.2.8.1)
> ==15979==    by 0x5D51B51: (within 
> /usr/lib/gtk-2.0/2.4.0/loaders/svg_loader.so)
> ==15979==    by 0x32871F: (within /usr/lib/libgdk_pixbuf-2.0.so.0.400.13)
> ==15979==    by 0x32885E: gdk_pixbuf_new_from_file (in 
> /usr/lib/libgdk_pixbuf-2.0.so.0.400.13)
> ==15979==    by 0x5CEE34: (within /usr/lib/libgtk-x11-2.0.so.0.400.13)
> ==15979==    by 0x5CF095: gtk_window_set_default_icon_from_file (in 
> /usr/lib/libgtk-x11-2.0.so.0.400.13)
> ==15979==    by 0x59FCD1B: _wrap_gtk_window_set_default_icon_from_file 
> (gtk.c:38156)
> ==15979==    by 0x80C29B6: PyEval_EvalFrameEx (ceval.c:3564)
> 
> For finding cycles that result in uncollectable garbage, I wrote a 
> cycle-finding utility (see attached file).  (I plan to submit this as a 
> Python Cookbook entry).  Given a list of objects of interest, it will 
> print out all the reference cycles involving those objects (though it 
> can't traverse through extension objects that don't expose reference 
> information to the garbage collector).
> 
> Objects that leak because legitimate Python references to them are still 
> around are some of the trickiest to find.  (This was an answer to a job 
> interview question I was once asked: "How can you leak memory in 
> Java/some other garbage collected language?")  I find it useful to 
> compare snapshots of all the objects in the interpreter before and after 
> a suspected leak:
> 
>    existing_objects = [id(x) for x in gc.get_objects()]
>    # ... do something that leaks ...
>    # ... delete everything you can ...
>    remaining_objects = [x for x in gc.get_objects() if id(x) not in 
> existing_objects]
> 
> One can then scour through remaining_objects for anything that suspected 
> of causing the problem, and do cycle detection on those objects, if 
> necessary to find ways to forcibly remove them.
> 
> Cheers,
> Mike
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel