Menu
▾
▴
matplotlib-users
|
From: A B <pyt...@gm...> - 2009-02-08 18:04:07
|
Hi,
Following is my post to the Django mailing list from yesterday. The response
was that Django isn't known to leak memory so there should be something off
with matplotlib or rather that way I am using it.
Hopefully someone here could comment on what could be causing the leaks.
Thanks in advance.
----
I am using matplotlib/pyplot on my site to dynamically generate PNG
plots. And I am experiencing dramatic memory leaks. Within 10-15
hits, my Apache process grows from 15-20M to 100M.
I am using Django 1.0.2-final, Apache 2.2.1, Python 2.4.3, matplotlib
0.98.5.2. The leak happens under both Apache (with mod_wsgi 2.3) and
the development server. My OS is RHEL5.
Below is a simple code snippet that causes the leak. Please let me
know if I am doing something wrong or if there is a better way to
write this. Thanks.
from matplotlib import pyplot
def test_graph (request):
f = pyplot.figure()
ax = f.add_subplot(111)
ax.plot([1,2,3])
ax.fill_between([1,2,3],[1,2,3],[1.1,2.1,3.1])
ax.grid(True)
ax.legend(['hello'],
'upper right', shadow=True, fancybox=True)
ax.set_xlabel('Time')
ax.set_ylabel('Value ')
f.text(.5, 0.93, 'my title', horizontalalignment='center')
response = HttpResponse(content_type='image/png')
### both ways causes a leak
f.savefig( response, format = 'png' )
OR
canvas = FigureCanvas(f)
canvas.print_png(response)
canvas = None
ax = None
f = None
return response
|
|
From: Jouni K. S. <jk...@ik...> - 2009-02-08 18:22:00
|
A B <pyt...@gm...> writes: > f = pyplot.figure() When you do this, matplotlib retains a reference to the figure until you close it so that you can go back to it with e.g. figure(4). So add pyplot.close(f) to your script. Or, even better, use the object-oriented API. To get started with that, see http://matplotlib.sourceforge.net/leftwich_tut.txt -- Jouni K. Seppänen http://www.iki.fi/jks |
|
From: A B <pyt...@gm...> - 2009-02-09 20:37:12
|
Thank you very much, Jouni. Pyplot.close() seems to have done the trick. I followed your advice and converted my code to use the OO interface. But I'm not sure how I close the figure in OO ... Any pointers? Thanks. |
|
From: Ryan M. <rm...@gm...> - 2009-02-09 22:54:28
|
On Mon, Feb 9, 2009 at 2:37 PM, A B <pyt...@gm...> wrote: > Thank you very much, Jouni. Pyplot.close() seems to have done the > trick. I followed your advice and converted my code to use the OO > interface. But I'm not sure how I close the figure in OO ... Any > pointers? Thanks. If you're using the full OO interface and creating a figure by making an instance of Figure(), then you shouldn't need to close the figure by hand. It should be deleted whenever you delete or replace your instance of Figure. (If I understand correctly.) Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma |
|
From: Jouni K. S. <jk...@ik...> - 2009-02-10 05:59:29
|
Ryan May <rm...@gm...> writes: > On Mon, Feb 9, 2009 at 2:37 PM, A B <pyt...@gm...> wrote: > > If you're using the full OO interface and creating a figure by making an > instance of Figure(), then you shouldn't need to close the figure by hand. > It should be deleted whenever you delete or replace your instance of > Figure. (If I understand correctly.) Yes, in the OO interface there is no close() because matplotlib does not retain any references to the figure object. Garbage collection will take care of reclaiming memory once the user code has no more references to the object either. That said, there have been cases of memory leaks caused by circular references among objects that have __del__ methods. I think all known leaks have been fixed, but if I were deploying a long-lived application that creates lots of figures, I would definitely want to watch its memory usage in my exact use case. -- Jouni K. Seppänen http://www.iki.fi/jks |
|
From: Gerry S. <ger...@gm...> - 2009-02-10 12:40:36
Attachments:
memleaktest.py
|
>> instance of Figure(), then you shouldn't need to close the figure by hand. >> It should be deleted whenever you delete or replace your instance of >> Figure. (If I understand correctly.) >... Garbage collection will take > care of reclaiming memory once the user code has no more references to > the object either. I guess that is how it should work in theory. However at least for the version of mpl shipped with ubuntu 8.10 (Version: 0.98.3-4ubuntu1) there is surely quite a serious and easily demonstrated memory leak using the artist api. Even if you manually del the figure objects which are created inside a loop a significant growth of memory can be observed. In fact I think there may be two separate leaks. In the attatched standalone code example after 1000 iterations creating 1000 png graphs there is around 2GB of RAM consumed. If you takeout the call to savefig around 600MB is consumed. In my view the code in the loop should execute in constant space without growth at all. Note that when the function returns the memory seems to be deallocated so if you check the ram use during the sleep you should see this. Any ideas on this? -- Gerry Steele http://belfast.no-ip.info/ |
|
From: A B <pyt...@gm...> - 2009-02-11 04:12:17
|
On Mon, Feb 9, 2009 at 9:59 PM, Jouni K. Seppänen <jk...@ik...> wrote: > Ryan May <rm...@gm...> writes: > > > On Mon, Feb 9, 2009 at 2:37 PM, A B <pyt...@gm...> wrote: > > > > If you're using the full OO interface and creating a figure by making an > > instance of Figure(), then you shouldn't need to close the figure by > hand. > > It should be deleted whenever you delete or replace your instance of > > Figure. (If I understand correctly.) > > Yes, in the OO interface there is no close() because matplotlib does not > retain any references to the figure object. Garbage collection will take > care of reclaiming memory once the user code has no more references to > the object either. > > That said, there have been cases of memory leaks caused by circular > references among objects that have __del__ methods. I think all known > leaks have been fixed, but if I were deploying a long-lived application > that creates lots of figures, I would definitely want to watch its > memory usage in my exact use case. > > -- > Jouni K. Seppänen > http://www.iki.fi/jks > Thank you. The reason I asked the question was that with the OO interface, I still observed growth in memory consumption (more than I saw after I added pyplot.close() calls). However, at some point (90M which I still think is a lot) the growth tapered off and eventually seemed to have stopped. Essentially, what I do is create a figure, then call plot() on it in a loop to generate multiple curves, then print_png() on the canvas object. |
|
From: Michael D. <md...@st...> - 2009-02-10 13:25:03
|
This is an instance of the OP's problem again. Your example is using the pyplot (i.e. Matlab-like) interface, in which case, you must explicitly close each figure when you're done with it, like follows: plt.close(fig) "del fig" only deletes your local reference to the figure. There is still a reference being held by the figure manager in case you wanted to go back to an earlier figure (accessed by number) and edit it. This is essentially the price paid for Matlab-work-a-likeness. I've tested your script on 0.98.3 (though on a RHEL4 system) and I don't see any memory leaks after adding the call to "plt.close(fig)". If you instantiate the figure and canvas directly (as in the agg_oo.py example), you won't have this issue. Mike Gerry Steele wrote: >>> instance of Figure(), then you shouldn't need to close the figure by hand. >>> It should be deleted whenever you delete or replace your instance of >>> Figure. (If I understand correctly.) >>> > > >> ... Garbage collection will take >> care of reclaiming memory once the user code has no more references to >> the object either. >> > > > I guess that is how it should work in theory. However at least for the > version of mpl shipped with ubuntu 8.10 (Version: 0.98.3-4ubuntu1) > there is surely quite a serious and easily demonstrated memory leak > using the artist api. Even if you manually del the figure objects > which are created inside a loop a significant growth of memory can be > observed. In fact I think there may be two separate leaks. > > In the attatched standalone code example after 1000 iterations > creating 1000 png graphs there is around 2GB of RAM consumed. If you > takeout the call to savefig around 600MB is consumed. > > In my view the code in the loop should execute in constant space > without growth at all. Note that when the function returns the memory > seems to be deallocated so if you check the ram use during the sleep > you should see this. > > Any ideas on this? > > > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) > software. With Adobe AIR, Ajax developers can use existing skills and code to > build responsive, highly engaging applications that combine the power of local > resources and data with the reach of the web. Download the Adobe AIR SDK and > Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com > ------------------------------------------------------------------------ > > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA |
|
From: Gerry S. <ger...@gm...> - 2009-02-10 13:55:17
|
Thanks Michael , I had somehow put myself under the impression i was using he OO version of the api but it is much more clear now. Memory issues now look better. Thanks. 2009/2/10 Michael Droettboom <md...@st...>: > This is an instance of the OP's problem again. Your example is using the > pyplot (i.e. Matlab-like) interface, in which case, you must explicitly > close each figure when you're done with it, like follows: > > plt.close(fig) > > "del fig" only deletes your local reference to the figure. There is still a > reference being held by the figure manager in case you wanted to go back to > an earlier figure (accessed by number) and edit it. This is essentially the > price paid for Matlab-work-a-likeness. > > I've tested your script on 0.98.3 (though on a RHEL4 system) and I don't see > any memory leaks after adding the call to "plt.close(fig)". > > If you instantiate the figure and canvas directly (as in the agg_oo.py > example), you won't have this issue. > > Mike > > Gerry Steele wrote: >>>> >>>> instance of Figure(), then you shouldn't need to close the figure by >>>> hand. >>>> It should be deleted whenever you delete or replace your instance of >>>> Figure. (If I understand correctly.) >>>> >> >> >>> >>> ... Garbage collection will take >>> care of reclaiming memory once the user code has no more references to >>> the object either. >>> >> >> >> I guess that is how it should work in theory. However at least for the >> version of mpl shipped with ubuntu 8.10 (Version: 0.98.3-4ubuntu1) >> there is surely quite a serious and easily demonstrated memory leak >> using the artist api. Even if you manually del the figure objects >> which are created inside a loop a significant growth of memory can be >> observed. In fact I think there may be two separate leaks. >> >> In the attatched standalone code example after 1000 iterations >> creating 1000 png graphs there is around 2GB of RAM consumed. If you >> takeout the call to savefig around 600MB is consumed. >> >> In my view the code in the loop should execute in constant space >> without growth at all. Note that when the function returns the memory >> seems to be deallocated so if you check the ram use during the sleep >> you should see this. >> >> Any ideas on this? >> >> >> >> ------------------------------------------------------------------------ >> >> >> ------------------------------------------------------------------------------ >> Create and Deploy Rich Internet Apps outside the browser with >> Adobe(R)AIR(TM) >> software. With Adobe AIR, Ajax developers can use existing skills and code >> to >> build responsive, highly engaging applications that combine the power of >> local >> resources and data with the reach of the web. Download the Adobe AIR SDK >> and >> Ajax docs to start building applications >> today-http://p.sf.net/sfu/adobe-com >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Matplotlib-users mailing list >> Mat...@li... >> https://lists.sourceforge.net/lists/listinfo/matplotlib-users >> > > -- > Michael Droettboom > Science Software Branch > Operations and Engineering Division > Space Telescope Science Institute > Operated by AURA for NASA > > -- Gerry Steele http://belfast.no-ip.info/ |
|
From: John H. <jd...@gm...> - 2009-02-10 14:05:45
|
On Tue, Feb 10, 2009 at 7:55 AM, Gerry Steele <ger...@gm...> wrote:
> Thanks Michael ,
>
> I had somehow put myself under the impression i was using he OO
> version of the api but it is much more clear now. Memory issues now
> look better.
There is room for confusion. A common usage pattern, one I often use
myself, is to use pyplot for figure creation, closing and showing
only, and to use the OO API for everything else. To whit:
import matplotlib.pyplot as plt
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot([1,2,3])
ax.set_xlabel('hi')
plt.show()
this contrasts with the pure pyplot approach, which uses the pyplot
state machine for everything (figure/axes creation, current axes, etc)
import matplotlib.pyplot as plt
plt.plot([1,2,3])
plt.xlabel('hi')
plt.show()
and both are different from the pure OO approach in which you manually
create the figure canvas etc
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
fig = Figure()
canvas = FigureCanvas(fig)
ax = fig.add_subplot(111)
ax.plot([1,2,3])
ax.set_xlabel('hi')
Note that in the 3rd example, all the code is the same as the first
example, except the figure/canvas creation, which is why one might
call it "OO pyplot"
Eric has summarized some additional information about the different
usage modes here
http://matplotlib.sourceforge.net/faq/usage_faq.html#matplotlib-pylab-and-pyplot-how-are-they-related
JDH
|
