Re: [Matplotlib-users] Matplotlib eating memory

[Martin M. <mmo...@gm...>]

Benjamin Root wrote:
> 
> 
> 
> On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmo...@gm... <mailto:mmo...@gm...>> wrote:
> 
>     Hi,
>       rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
>     of one such situation when it already took 15GB. Would somebody comments on what is
>     matplotlib doing at the very moment? Why the recursion?
> 
>       The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
>     its own color. They are in batches so that there are 153 distinct colors but nevertheless,
>     I assigned to each data point a color value. There are 153 legend items also (one color
>     won't be used).
> 
>     ^CTraceback (most recent call last):
>     ...
>         _figure.savefig(filename, dpi=100)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
>         self.canvas.print_figure(*args, **kwargs)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
>         **kwargs)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
>         FigureCanvasAgg.draw(self)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
>         self.figure.draw(self.renderer)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>         draw(artist, renderer, *args, **kwargs)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
>         func(*args)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>         draw(artist, renderer, *args, **kwargs)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
>         a.draw(renderer)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>         draw(artist, renderer, *args, **kwargs)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
>         return Collection.draw(self, renderer)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>         draw(artist, renderer, *args, **kwargs)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
>         offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
>       File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
>         return self._edgecolors
>     KeyboardInterrupt
>     ^CError in atexit._run_exitfuncs:
>     Traceback (most recent call last):
>       File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>         func(*targs, **kargs)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>         gc.collect()
>     KeyboardInterrupt
>     Error in sys.exitfunc:
>     Traceback (most recent call last):
>       File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>         func(*targs, **kargs)
>       File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>         gc.collect()
>     KeyboardInterrupt
> 
>     ^C
> 
> 
>     Clues what is the code doing? I use mpl-1.3.0.
>     Thank you,
>     Martin
> 
> 
> Unfortunately, that stacktrace isn't very useful. There is no recursion there, but rather the perfectly normal drawing of the figure object that has a child axes, which has child collections which have child artist objects.
> 
> Without the accompanying code, it would be difficult to determine where the memory hog is.

Could there be places where gc.collect() could be introduced? Are there places where matplotlib
could del() unnecessary objects right away? I think the problem is with huge lists or pythonic
dicts. I could save 10GB of RAM when I converted one python dict to a bsddb3 file having just
10MB on disk. I speculate matplotlib in that code keeps the data in some huge list or more likely
a dict and that is the same issue.

Are you sure you cannot see where a problem is? It happens (is visible) only with huge number of
dots, of course.

Thanks,
Martin