From: Steve C. <ste...@ya...> - 2004-06-08 09:53:44
|
There does seem to be a memory problem when using Python and GTK+. Have you seen the pygtk FAQ 8.4 "Is there a resource leak? Why do I run out of memory using Pixbuf?" >From looking at matplotlib/backends/backend_gtk.py I noticed every FigureCanvasGTK.draw() operation allocates memory for a new pixmap and a new graphics context. So looping 200 times in your first test script allocates memory for 200 pixmaps and 200 graphics contexts, yet the pixmap size has not changed and the graphics context has not changed. I think it would be more memory efficient to create a pixmap and continue using it until a larger pixmap is required. The graphics context does not change its default settings, so it does little, but is required for the GDK calls, again it looks to me like it would be better to create one and reuse it. Here are some changes I made to matplotlib/backends/backend_gtk.py I estimate saves around 10% of the memory leak. However I can't check that it plots correctly properly because I currently get a plot of a black rectangle in GTK mode (GTKAgg works OK), which I think was mentioned as a numarray problem with the latest matplotlib release. FigureCanvasGTK.__init__(self,figure): # add the following line self.pixmap_width, self.pixmap_height = -1, -1 # new FigureCanvasGTK.draw(self): # replace with this: def draw(self): if not self._doplot: return drawable = self.window if drawable is None: return width = int(self.figure.bbox.width()) height = int(self.figure.bbox.height()) if width > self.pixmap_width or height > self.pixmap_height: self._gpixmap = gtk.gdk.Pixmap(drawable, width, height) self.pixmap_width = width self.pixmap_height = height self.figure.draw( RendererGTK(self, self._gpixmap, self.figure.dpi) ) drawable.draw_drawable(self._ggc, self._gpixmap, 0, 0, 0, 0, width, height) FigureCanvasGTK.realize(self, widget): # add the following line: self._ggc = self.window.new_gc() Regards Steve |
From: Eric F. <ef...@km...> - 2004-06-09 03:42:14
|
Steve, >There does seem to be a memory problem when using Python and GTK+. Have >you seen the pygtk FAQ 8.4 "Is there a resource leak? Why do I run out >of memory using Pixbuf?" > > > Yes, I saw that, tried putting in explicit gc.collect() calls, but it didn't help. I concluded the problem was different. I also looked at the draw function in backend_gtk.py. What I don't understand is how and when the no-longer-used gtk objects (pixmap, etc) really can and should be de-allocated. Evidently they persist when the python reference to them goes out of scope. Thanks for your changes. Eric >>From looking at matplotlib/backends/backend_gtk.py I noticed every >FigureCanvasGTK.draw() operation allocates memory for a new pixmap and a >new graphics context. So looping 200 times in your first test script >allocates memory for 200 pixmaps and 200 graphics contexts, yet the >pixmap size has not changed and the graphics context has not changed. I >think it would be more memory efficient to create a pixmap and continue >using it until a larger pixmap is required. The graphics context does >not change its default settings, so it does little, but is required for >the GDK calls, again it looks to me like it would be better to create >one and reuse it. > >Here are some changes I made to matplotlib/backends/backend_gtk.py I >estimate saves around 10% of the memory leak. >However I can't check that it plots correctly properly because I >currently get a plot of a black rectangle in GTK mode (GTKAgg works OK), >which I think was mentioned as a numarray problem with the latest >matplotlib release. > >FigureCanvasGTK.__init__(self,figure): # add the following line > self.pixmap_width, self.pixmap_height = -1, -1 # new > > > FigureCanvasGTK.draw(self): # replace with this: > def draw(self): > if not self._doplot: return > drawable = self.window > if drawable is None: return > > > > width = int(self.figure.bbox.width()) > height = int(self.figure.bbox.height()) > > if width > self.pixmap_width or height > self.pixmap_height: > self._gpixmap = gtk.gdk.Pixmap(drawable, width, height) > self.pixmap_width = width > self.pixmap_height = height > > self.figure.draw( RendererGTK(self, self._gpixmap, >self.figure.dpi) ) > drawable.draw_drawable(self._ggc, self._gpixmap, 0, 0, 0, 0, >width, height) > >FigureCanvasGTK.realize(self, widget): # add the following line: > self._ggc = self.window.new_gc() > >Regards >Steve > > > > |
From: John H. <jdh...@ac...> - 2004-06-09 22:16:34
|
>>>>> "Eric" == Eric Firing <ef...@km...> writes: Eric> Yes, I saw that, tried putting in explicit gc.collect() Eric> calls, but it didn't help. I concluded the problem was Eric> different. I also looked at the draw function in Eric> backend_gtk.py. What I don't understand is how and when the Eric> no-longer-used gtk objects (pixmap, etc) really can and Eric> should be de-allocated. Evidently they persist when the Eric> python reference to them goes out of scope. Eric> Thanks for your changes. Hi Eric, You can test the new release which includes both Steve's changes and a number of fixes on my end. I'll attach the script below that I used to profile the leak, a modified version of your script. By my numbers, the average memory consumed per loop is down from 145.9 using 0.54.1 to 12.3 using 0.54.2. Not perfect, but more than 10x better. I have definitely identified a leak in freetype (as mentioned above, which is fixed in freetype CVS) and there appears to be a leak in libpng but I have tracked down precisely where. Neither of these are terribly large. I strongly suspect that some of the remaining leak is my doing, so I'll continue to try and sniff these out. The python code uses a lot of circular references (figures contain axes and text which in turn contain a reference to the figure that contains them and so on) which makes the task a little harder. Let me know how it goes. JDH import os, sys, time import matplotlib matplotlib.use('Agg') from matplotlib.matlab import * def report_memory(i): pid = os.getpid() a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines() print i, ' ', a2[1], return int(a2[1].split()[0]) def updatefig(i): ind = arange(100) xx = rand(len(ind), 1) figure(1) plot(ind, xx[:,0]) savefig('dd_ef%d.png' % i, dpi = 75) close(1) return report_memory(i) N = 100 for i in range(N): val = updatefig(i) if i==1: start = val end = val print 'Average memory consumed per loop: %1.4f\n' % ((end-start)/float(N)) |
From: Steve C. <ste...@ya...> - 2004-06-11 08:33:09
|
On Thu, 2004-06-10 at 05:52, John Hunter wrote: > Hi Eric, > > You can test the new release which includes both Steve's changes and a > number of fixes on my end. I'll attach the script below that I used > to profile the leak, a modified version of your script. By my > numbers, the average memory consumed per loop is down from 145.9 using > 0.54.1 to 12.3 using 0.54.2. Not perfect, but more than 10x better. > > I have definitely identified a leak in freetype (as mentioned above, > which is fixed in freetype CVS) and there appears to be a leak in > libpng but I have tracked down precisely where. Neither of these are > terribly large. I strongly suspect that some of the remaining leak is > my doing, so I'll continue to try and sniff these out. The python > code uses a lot of circular references (figures contain axes and text > which in turn contain a reference to the figure that contains them and > so on) which makes the task a little harder. > > Let me know how it goes. > > JDH I ran your memory test script on my Fedora 2 PC, it gave Average memory consumed per loop: 168.3600 on the previous matplotlib Average memory consumed per loop: 41.7200 on the latest matplotlib from CVS. That's a big improvement. For the last week or so I've been unable to use the GTK+ backend with matplotlib (and PyGTK) from CVS. Instead of producing a plot it colors the whole figure black. I think this may be a colormap problem. Steve |
From: John H. <jdh...@ac...> - 2004-06-11 11:46:37
|
>>>>> "Steve" == Steve Chaplin <ste...@ya...> writes: Steve> I ran your memory test script on my Fedora 2 PC, it gave Steve> Average memory consumed per loop: 168.3600 on the previous Steve> matplotlib Average memory consumed per loop: 41.7200 on the Steve> latest matplotlib from CVS. That's a big improvement. Well that's interesting. I just reran the script myself because I was getting approx 12 per loop and you are getting 40. But now when I rerun it I get numbers compatible with yours. Annoying. Back to the drawing board. Steve> For the last week or so I've been unable to use the GTK+ Steve> backend with matplotlib (and PyGTK) from CVS. Instead of Steve> producing a plot it colors the whole figure black. I think Steve> this may be a colormap problem. From your post to the pygtk list, I assume you are using PyGTK 2.3.93. Unfortunately I don't have a good platform to test this version on since my system doesn't have the required glib/gtk version and I have learned from past experience not to try and upgrade these libs. 0.54.2 with the GTK backend runs fine with pygtk-2.2 on my system. A couple of questions * can you run matplotlib-0.54.1 on your current version of pygtk. This will help me figure out if something in matplotlib has changed or if something on your system has changed. The GTK backend is fairly stable at this point so not a lot has changed. * Can you run matplotlib-0.54.2 with pygtk2.2 on your system? If there is a 2.3.93 specific problem, we'll need to get it worked out, but I'd like to narrow the field of candidate problems first. JDH |
From: Steve C. <ste...@ya...> - 2004-06-11 13:24:21
|
On Fri, 2004-06-11 at 19:22, John Hunter wrote: > Steve> For the last week or so I've been unable to use the GTK+ > Steve> backend with matplotlib (and PyGTK) from CVS. Instead of > Steve> producing a plot it colors the whole figure black. I think > Steve> this may be a colormap problem. > > >From your post to the pygtk list, I assume you are using PyGTK 2.3.93. > Unfortunately I don't have a good platform to test this version on > since my system doesn't have the required glib/gtk version and I have > learned from past experience not to try and upgrade these libs. > 0.54.2 with the GTK backend runs fine with pygtk-2.2 on my system. > > A couple of questions > > * can you run matplotlib-0.54.1 on your current version of pygtk. > This will help me figure out if something in matplotlib has changed > or if something on your system has changed. The GTK backend is > fairly stable at this point so not a lot has changed. > > * Can you run matplotlib-0.54.2 with pygtk2.2 on your system? > > If there is a 2.3.93 specific problem, we'll need to get it worked > out, but I'd like to narrow the field of candidate problems first. > > JDH I've just ran lots of tests: Matplotlib PyGTK GTK+ backend ---------- ----- ------------ 0.54.1 2.2.0 working 0.54.2 2.2.0 working cvs 2.2.0 working 0.54.1 cvs(2.3.93) fails (the figure is all black) 0.54.2 cvs(2.3.93) fails cvs cvs(2.3.93) fails It must be a PyGTK bug. Strangely enough I opened a PyGTK bug report today (#144135), which turns out to be serious bug - a 'stopper' for PyGTK 2.4. It may also be causing this problem with matplotlib. Hopefully it will be fixed soon and if it doesn't fix this problem running matplotlib I'll open another PyGTK bug report. Steve |
From: John H. <jdh...@ac...> - 2004-06-08 14:17:48
|
>>>>> "Steve" == Steve Chaplin <ste...@ya...> writes: Steve> There does seem to be a memory problem when using Python Steve> and GTK+. Have you seen the pygtk FAQ 8.4 "Is there a Steve> resource leak? Why do I run out of memory using Pixbuf?" I submitted a patch to fix a pygtk memory leak in get_image that was incorporated into pygtk 2.2. http://bugzilla.gnome.org/show_bug.cgi?id=133681 Which version are you using? Do you still see a problem with pygtk 2.2 or later? I'll take a look at incorporating your changes, in an case. But as Kirill pointed out, there is a separate problem with Agg that I have to track down. JDH |
From: Eric F. <ef...@km...> - 2004-06-09 03:42:16
|
John, I am using pygtk2.2, built and installed from the tarball on top of my existing pygtk2.0, which was installed from rpm. The problem does not seem to be inevitable with pygtk plotting; testing I have done so far with pyStripchart 0.0.7 (http://jstripchart.sourceforge.net), also with pygtk2.2, has given no indication of a leak. It is a much more specialized (hence simpler) package, though. It draws directly to a DrawingArea instead of using a Pixmap. Eric John Hunter wrote: >>>>>>"Steve" == Steve Chaplin <ste...@ya...> writes: >>>>>> >>>>>> > > Steve> There does seem to be a memory problem when using Python > Steve> and GTK+. Have you seen the pygtk FAQ 8.4 "Is there a > Steve> resource leak? Why do I run out of memory using Pixbuf?" > >I submitted a patch to fix a pygtk memory leak in get_image that was >incorporated into pygtk 2.2. > > http://bugzilla.gnome.org/show_bug.cgi?id=133681 > >Which version are you using? Do you still see a problem with pygtk >2.2 or later? > >I'll take a look at incorporating your changes, in an case. > >But as Kirill pointed out, there is a separate problem with Agg that I >have to track down. > >JDH > > > |
From: John H. <jdh...@ac...> - 2004-06-09 13:06:54
|
>>>>> "Eric" == Eric Firing <ef...@km...> writes: Eric> John, I am using pygtk2.2, built and installed from the Eric> tarball on top of my existing pygtk2.0, which was installed Eric> from rpm. Eric> The problem does not seem to be inevitable with pygtk Eric> plotting; testing I have done so far with pyStripchart 0.0.7 Eric> (http://jstripchart.sourceforge.net), also with pygtk2.2, Eric> has given no indication of a leak. It is a much more Eric> specialized (hence simpler) package, though. It draws Eric> directly to a DrawingArea instead of using a Pixmap. Hi Eric, I spent a good bit of time working on the memory leak problem. It turns out there were numerous problems in both the transforms module and the ft2font module, which affect all backends. I've got the former entirely cleared up, and the latter almost done. I'm writing some unit tests to check for leaks as I go. In the next day or so I should have a snapshot ready for you to test. So while there may still be a small problem with pygtk >2.0, I'm not aware of it. I think the major problems rest squarely on the shoulders of yours truly. I wrote the transforms module using CXX, which is a C++ python extension building library that handles most of the nasty bits for you, including reference counting. But I was too reliant on it, and there were some subtleties that I was neglecting involved with managing reference counts when new pointers are allocated on the heap and stored in other classes. Fortunately it was a fairly easy fix and now the transforms are not leaking any detectable memory. The ft2font module was another matter, since I had written that by hand. I decided the best way to get this in tip-top shape was to also rewrite this using CXX, rather than work out all of the intricacies of reference counting myself. That was a bit more work, but I'm mostly done. I found one very small leak in the freetype library itself, but a little searching on the devel list shows that it is fixed in CVS >2.1.8. But this is only a few bytes per font instance (which are cached and reused by matplotlib) so this shouldn't present any troubles. When I get everything cleaned up I'll put together some before and after numbers from my unit tests. Hope all is well at sea! JDH |
From: Eric F. <ef...@km...> - 2004-06-10 00:14:49
|
John, Thanks very much. If you would like me to try the new snapshot, and it is ready soon enough, let me know what the email attachment size would be, if it would exceed about 100k. There is a limit on email size going to the ship, but it can be relaxed if I know in advance that something is coming. I have encountered surprisingly large differences between Numeric and numarray: pcolor_demo2.py takes about 3 seconds with Numeric, 80 seconds with numarray. In tests of non-plotting operations, I have generally found numarray faster. The cruise is going fine. Weather was a bit rough off New Zealand but I expect it will be fairly smooth from here on. This is a Small Waterplane Area Twin Hull (SWATH) ship, so it rides better than most; it is a little like a catamaran on stilts, with most of the buoyancy in fully submerged pontoons. Eric ... > >When I get everything cleaned up I'll put together some before and >after numbers from my unit tests. > >Hope all is well at sea! > >JDH > > > |
From: Steve C. <ste...@ya...> - 2004-06-09 06:07:38
|
On Tue, 2004-06-08 at 21:53, John Hunter wrote: > >>>>> "Steve" == Steve Chaplin <ste...@ya...> writes: > > Steve> There does seem to be a memory problem when using Python > Steve> and GTK+. Have you seen the pygtk FAQ 8.4 "Is there a > Steve> resource leak? Why do I run out of memory using Pixbuf?" > > I submitted a patch to fix a pygtk memory leak in get_image that was > incorporated into pygtk 2.2. > > http://bugzilla.gnome.org/show_bug.cgi?id=133681 > > Which version are you using? Do you still see a problem with pygtk > 2.2 or later? Thanks for the link. I'm using pygtk from cvs and ran a test for myself to check that get_image() no longer leaks memory. I think the FAQ 8.4 reports a memory inefficiency problem, not a memory leak. It shows a loop where you quickly allocate memory and can use a large amount of system memory before the Python garbage collector has a chance to free the unused memory. Steve |