matplotlib-users

[Matplotlib-users] memory usage (leakage?) in ipython interactive mode

[Gary R. <gr...@bi...>]

Is there a summary somewhere of the current state of knowledge about 
memory leaks when using the pylab interface interactively? Doing 
plot(rand(1000000)) or matshow(rand(1000,1000)) for example eats a big 
chunk of memory (tried with TkAgg and WxAgg in Windows (mpl v0.98.5.2) 
and Linux (mpl v0.98.3)), most of which is not returned when the window 
is closed. The same goes if you create an array, plot it, and explicitly 
del it after closing the window. I've seen lots of posts over the years 
about memory leaks, but there's nothing in the FAQ about this. I found 
old posts about similar things, but nothing that had a clear resolution.

thanks,
Gary

Re: [Matplotlib-users] memory usage (leakage?) in ipython interactive mode

[Michael D. <md...@st...>]

There are at least three possible causes of what you're seeing here:

1) ipython stores references to all results in the console.  (ipython 
maintains a history of results so they can easily be accessed later).  I 
don't recall the details, but it may be possible to turn this feature 
off or limit the number of objects stored.

2) matplotlib stores references to all figures until they are explicitly 
closed with pyplot.close(fignum)

3) Python uses pools of memory, and is often imposes a significant delay 
returning memory to the operating system.  It is actually very hard to 
determine from the outside whether something is leaking or just pooling 
without compiling a special build of Python with memory pooling turned off.

In general, interactive use is somewhat at odds with creating many large 
plots in a single session, since all of the nice interactive features 
(history etc.) do not know automagically when the user is done with 
certain objects.

I am not aware of any memory leaks in current versions of matplotlib 
with *noninteractive* use, other than small leaks caused by bugs in 
older versions of some of the GUI toolkits (notably gtk+).  If you find 
a script that produces a leak reproducibly, please share so we can track 
down the cause.

Gary Ruben wrote:
> Doing 
> plot(rand(1000000)) or matshow(rand(1000,1000)) for example eats a big 
> chunk of memory (tried with TkAgg and WxAgg in Windows (mpl v0.98.5.2) 
> and Linux (mpl v0.98.3)), most of which is not returned when the window 
> is closed. The same goes if you create an array, plot it, and explicitly 
> del it after closing the window.
Can you elaborate on these steps?  It's possible that the del has little 
effect, since del only deletes a single reference to the object, not all 
references which may be keeping it alive (such as the figure, which 
matplotlib itself keeps a reference to).  In general, you need to 
explicitly call pyplot.close(fignum) to delete a figure.

Cheers,
Mike

-- 
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Re: [Matplotlib-users] memory usage (leakage?) in ipython interactive mode

[Gary R. <gr...@bi...>]

Hi Michael,

Thanks for your explanation. It turns out that it is a combination of
(1) and (3). I hadn't thought about (1) and I hadn't done enough
playing to see the python interpreter releasing blocks of memory. As
you suggested, the "solution" is to limit the iPython cache by using
the iPython -cs option.

thanks for your help,
Gary

Michael Droettboom wrote:
> There are at least three possible causes of what you're seeing here:
> 
> 1) ipython stores references to all results in the console.  (ipython 
> maintains a history of results so they can easily be accessed later).  I 
> don't recall the details, but it may be possible to turn this feature 
> off or limit the number of objects stored.
> 
> 2) matplotlib stores references to all figures until they are explicitly 
> closed with pyplot.close(fignum)
> 
> 3) Python uses pools of memory, and is often imposes a significant delay 
> returning memory to the operating system.  It is actually very hard to 
> determine from the outside whether something is leaking or just pooling 
> without compiling a special build of Python with memory pooling turned off.
> 
> In general, interactive use is somewhat at odds with creating many large 
> plots in a single session, since all of the nice interactive features 
> (history etc.) do not know automagically when the user is done with 
> certain objects.
> 
> I am not aware of any memory leaks in current versions of matplotlib 
> with *noninteractive* use, other than small leaks caused by bugs in 
> older versions of some of the GUI toolkits (notably gtk+).  If you find 
> a script that produces a leak reproducibly, please share so we can track 
> down the cause.
> 
> Gary Ruben wrote:
>> Doing plot(rand(1000000)) or matshow(rand(1000,1000)) for example eats 
>> a big chunk of memory (tried with TkAgg and WxAgg in Windows (mpl 
>> v0.98.5.2) and Linux (mpl v0.98.3)), most of which is not returned 
>> when the window is closed. The same goes if you create an array, plot 
>> it, and explicitly del it after closing the window.
> Can you elaborate on these steps?  It's possible that the del has little 
> effect, since del only deletes a single reference to the object, not all 
> references which may be keeping it alive (such as the figure, which 
> matplotlib itself keeps a reference to).  In general, you need to 
> explicitly call pyplot.close(fignum) to delete a figure.
> 
> Cheers,
> Mike
>