Menu
▾
▴
Re: [Matplotlib-users] Possible memory leak?
|
From: Michael D. <md...@st...> - 2010-11-22 16:13:30
|
Caleb,
Thanks for doing all of this investigation and providing something easy
to reproduce.
With the help of valgrind, I believe I've tracked it down to a bug in
PyCXX, the Python/C++ interface tool matplotlib uses.
I have attached a patch that seems to remove the leak for me, but as I'm
not a PyCXX expert, I'm not comfortable with committing it to the
repository just yet. *I'm hoping you and/or some other developers could
test it on their systems (a fully clean re-build is required) and report
any problems back. *I also plan to raise this question on the PyCXX
mailing list to get any thoughts they may have.
Cheers,
Mike
On 11/19/2010 04:14 PM, Caleb Constantine wrote:
> On Thu, Nov 18, 2010 at 4:50 PM, Benjamin Root<ben...@ou...> wrote:
>> Caleb,
>>
>> Interesting analysis. One possible source of a leak would be some sort of dangling reference that still hangs around even though the plot objects have been cleared. By the time of the matplotlib 1.0.0 release, we did seem to clear out pretty much all of these, but it is possible there are still some lurking about. We should probably run your script against the latest svn to see how the results compare.
>>
>> Another possibility might be related to numpy. However this is the draw statement, so I don't know how much numpy is used in there. The latest refactor work in numpy has revealed some memory leaks that have existed, so who knows?
>>
>> Might be interesting to try making equivalent versions of this script using different backends, and different package versions to possibly isolate the source of the memory leak.
>>
>> Thanks for your observations,
>> Ben Root
>>
> Sorry for the double post; it seems the first is not displaying
> correctly on SourceForge.
>
> I conducted a couple more experiments taking into consideration suggestions
> made in responses to my original post (thanks for the response).
>
> First, I ran my original test (as close to it as possible anyway) using the
> Agg back end for 3 hours, plotting 16591 times (about 1.5Hz). Memory usage
> increased by 86MB. That's about 5.3K per redraw. Very similar to my original
> experiment. As suggested, I called gc.collect() after each iteration. It
> returned 67 for every iteration (no increase), although len(gc.garbage)
> reported 0 each iteration.
>
> Second, I ran a test targeting TkAgg for 3 hours, plotting 21374 times. Memory
> usage fluctuated over time, but essentially did not increase: starting at
> 32.54MB and ending at 32.79MB. gc.collect() reported 0 after each iteration
> as did len(gc.garbage).
>
> Attached are images of plots showing change in memory usage over time for each
> experiment.
>
> Any comments would be appreciated.
>
> Following is the code for each experiment.
>
> Agg
> -----
>
> from random import random
> from datetime import datetime
> import os
> import gc
> import time
> import win32api
> import win32con
> import win32process
>
> import numpy
>
> import matplotlib
> matplotlib.use("Agg")
> from matplotlib.figure import Figure
> from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
>
> def get_process_memory_info(process_id):
> memory = {}
> process = None
> try:
> process = win32api.OpenProcess(
> win32con.PROCESS_QUERY_INFORMATION|win32con.PROCESS_VM_READ,
> False, process_id);
> if process is not None:
> return win32process.GetProcessMemoryInfo(process)
> finally:
> if process:
> win32api.CloseHandle(process)
> return memory
>
> meg = 1024.0 * 1024.0
>
> figure = Figure(dpi=None)
> canvas = FigureCanvas(figure)
> axes = figure.add_subplot(1,1,1)
>
> def draw(channel, seconds):
> axes.clear()
> axes.plot(channel, seconds)
> canvas.print_figure('test.png')
>
> channel = numpy.sin(numpy.arange(1000) * random())
> seconds = numpy.arange(len(channel))
> testDuration = 60 * 60 * 3
> startTime = time.time()
>
> print "starting memory: ", \
> get_process_memory_info(os.getpid())["WorkingSetSize"]/meg
>
> while (time.time() - startTime)< testDuration:
> draw(channel, seconds)
>
> t = datetime.now()
> memory = get_process_memory_info(os.getpid())
> print "time: {0}, working: {1:f}, collect: {2}, garbage: {3}".format(
> t,
> memory["WorkingSetSize"]/meg,
> gc.collect(),
> len(gc.garbage) )
>
> time.sleep(0.5)
>
>
> TkAgg
> ---------
> from random import random
> from datetime import datetime
> import sys
> import os
> import gc
> import time
> import win32api
> import win32con
> import win32process
>
> import numpy
>
> import matplotlib
> matplotlib.use("TkAgg")
> from matplotlib.figure import Figure
> from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg \
> as FigureCanvas
>
> import Tkinter as tk
>
> def get_process_memory_info(process_id):
> memory = {}
> process = None
> try:
> process = win32api.OpenProcess(
> win32con.PROCESS_QUERY_INFORMATION|win32con.PROCESS_VM_READ,
> False, process_id);
> if process is not None:
> return win32process.GetProcessMemoryInfo(process)
> finally:
> if process:
> win32api.CloseHandle(process)
> return memory
>
> meg = 1024.0 * 1024.0
>
> rootTk = tk.Tk()
> rootTk.wm_title("TKAgg Memory Leak")
>
> figure = Figure()
> canvas = FigureCanvas(figure, master=rootTk)
> axes = figure.add_subplot(1,1,1)
>
> def draw(channel, seconds):
> axes.clear()
> axes.plot(channel, seconds)
>
> channel = numpy.sin(numpy.arange(1000) * random())
> seconds = numpy.arange(len(channel))
>
> testDuration = 60 * 60 * 3
> startTime = time.time()
>
> print "starting memory: ", \
> get_process_memory_info(os.getpid())["WorkingSetSize"]/meg
>
> draw(channel, seconds)
> canvas.show()
> canvas.get_tk_widget().pack(side=tk.TOP, fill=tk.BOTH, expand=1)
>
> rate = 500
>
> def on_tick():
> canvas.get_tk_widget().after(rate, on_tick)
>
> if (time.time() - startTime)>= testDuration:
> return
>
> draw(channel, seconds)
>
> t = datetime.now()
> memory = get_process_memory_info(os.getpid())
> print "time: {0}, working: {1:f}, collect: {2}, garbage: {3}".format(
> t,
> memory["WorkingSetSize"]/meg,
> gc.collect(),
> len(gc.garbage) )
>
> canvas.get_tk_widget().after(rate, on_tick)
> tk.mainloop()
>
>
> ------------------------------------------------------------------------------
> Beautiful is writing same markup. Internet Explorer 9 supports
> standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2& L3.
> Spend less time writing and rewriting code and more time creating great
> experiences on the web. Be a part of the beta today
> http://p.sf.net/sfu/msIE9-sfdev2dev
>
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
|
