matplotlib-users

[Matplotlib-users] Trouble with imshow

[Robert A. <ab...@ss...>]

Hello All,

I'm very new to python, so bear with me.

I'd like to use python to do my image processing, but I'm running into behavior that doesn't make 
sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest 
versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file 
that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB, 
but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each 
time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something 
wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result. 
Here's a transcript.

   Welcome to pylab, a matplotlib-based Python environment.
   For more information, type 'help(pylab)'.

In [1]: import pyfits

In [2]: from Tkinter import *

In [3]: import tkFileDialog

In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())

In [5]: imshow(image)
Out[5]: <matplotlib.image.AxesImage object at 0x03BCA170>

In [6]: close()

In [7]: imshow(image,origin='lower')
Out[7]: <matplotlib.image.AxesImage object at 0x0440E170>

In [8]: close()

In [9]: imshow(image[100:3600,100:3600],origin='lower')
Out[9]: <matplotlib.image.AxesImage object at 0x045D9FB0>

In [10]: Exception in Tkinter callback
Traceback (most recent call last):
   File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
     return self.func(*args)
   File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
     func(*args)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in 
idle_draw
     self.draw()
   File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
     FigureCanvasAgg.draw(self)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
     self.figure.draw(self.renderer)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
     draw(artist, renderer, *args, **kwargs)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
     func(*args)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
     draw(artist, renderer, *args, **kwargs)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
     a.draw(renderer)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
     draw(artist, renderer, *args, **kwargs)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
     im = self.make_image(renderer.get_image_magnification())
   File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
     transformed_viewLim)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
     x = self.to_rgba(self._A, self._alpha)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
     x = self.norm(x)
   File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
     result = (val-vmin) / (vmax-vmin)
   File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
     return divide(self, other)
   File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
     m |= filled(domain(da, db), True)
   File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
     return umath.absolute(a) * self.tolerance >= umath.absolute(b)
MemoryError


Thanks for any help,
-robert

Re: [Matplotlib-users] Trouble with imshow

[Benjamin R. <ben...@ou...>]

On Wed, Feb 2, 2011 at 5:33 PM, Robert Abiad <ab...@ss...> wrote:

> Hello All,
>
> I'm very new to python, so bear with me.
>
> I'd like to use python to do my image processing, but I'm running into
> behavior that doesn't make
> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory,
> python 2.6.6, and the newest
> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm
> loading in a fits file
> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see
> memory usage go up by 50MB,
> but when I try displaying the image using imshow(), my memory usage goes up
> by around 500MB, each
> time.  If I close the figure and replot it, imshow() crashes.  I don't know
> if I'm doing something
> wrong, or if it's a new or known bug.  I tried the same thing on Linux and
> got the same result.
> Here's a transcript.
>
>   Welcome to pylab, a matplotlib-based Python environment.
>   For more information, type 'help(pylab)'.
>
> In [1]: import pyfits
>
> In [2]: from Tkinter import *
>
> In [3]: import tkFileDialog
>
> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>
> In [5]: imshow(image)
> Out[5]: <matplotlib.image.AxesImage object at 0x03BCA170>
>
> In [6]: close()
>
> In [7]: imshow(image,origin='lower')
> Out[7]: <matplotlib.image.AxesImage object at 0x0440E170>
>
> In [8]: close()
>
> In [9]: imshow(image[100:3600,100:3600],origin='lower')
> Out[9]: <matplotlib.image.AxesImage object at 0x045D9FB0>
>
> In [10]: Exception in Tkinter callback
> Traceback (most recent call last):
>   File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>     return self.func(*args)
>   File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>     func(*args)
>   File
> "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py",
> line 263, in
> idle_draw
>     self.draw()
>   File
> "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py",
> line 248, in draw
>     FigureCanvasAgg.draw(self)
>   File
> "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 394, in draw
>     self.figure.draw(self.renderer)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55,
> in draw_wrapper
>     draw(artist, renderer, *args, **kwargs)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798,
> in draw
>     func(*args)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55,
> in draw_wrapper
>     draw(artist, renderer, *args, **kwargs)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946,
> in draw
>     a.draw(renderer)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55,
> in draw_wrapper
>     draw(artist, renderer, *args, **kwargs)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354,
> in draw
>     im = self.make_image(renderer.get_image_magnification())
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569,
> in make_image
>     transformed_viewLim)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201,
> in _get_unsampled_image
>     x = self.to_rgba(self._A, self._alpha)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in
> to_rgba
>     x = self.norm(x)
>   File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820,
> in __call__
>     result = (val-vmin) / (vmax-vmin)
>   File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in
> __div__
>     return divide(self, other)
>   File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in
> __call__
>     m |= filled(domain(da, db), True)
>   File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in
> __call__
>     return umath.absolute(a) * self.tolerance >= umath.absolute(b)
> MemoryError
>
>
> Thanks for any help,
> -robert
>
>
This isn't new, but it isn't well understood either:

http://www.mail-archive.com/mat...@li.../msg19614.html

Unfortunately, that conversation went off-list, and never resulted in a bug
report being filed.  Could you please file a bug report on this?

http://sourceforge.net/tracker/?group_id=80706

Thanks,
Ben Root

Re: [Matplotlib-users] Trouble with imshow

[Christoph G. <cg...@uc...>]

On 2/2/2011 3:33 PM, Robert Abiad wrote:
> Hello All,
>
> I'm very new to python, so bear with me.
>
> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
> Here's a transcript.
>
>     Welcome to pylab, a matplotlib-based Python environment.
>     For more information, type 'help(pylab)'.
>
> In [1]: import pyfits
>
> In [2]: from Tkinter import *
>
> In [3]: import tkFileDialog
>
> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>
> In [5]: imshow(image)
> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>
> In [6]: close()
>
> In [7]: imshow(image,origin='lower')
> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>
> In [8]: close()
>
> In [9]: imshow(image[100:3600,100:3600],origin='lower')
> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>
> In [10]: Exception in Tkinter callback
> Traceback (most recent call last):
>     File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>       return self.func(*args)
>     File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>       func(*args)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
> idle_draw
>       self.draw()
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>       FigureCanvasAgg.draw(self)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>       self.figure.draw(self.renderer)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>       draw(artist, renderer, *args, **kwargs)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>       func(*args)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>       draw(artist, renderer, *args, **kwargs)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>       a.draw(renderer)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>       draw(artist, renderer, *args, **kwargs)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>       im = self.make_image(renderer.get_image_magnification())
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>       transformed_viewLim)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>       x = self.to_rgba(self._A, self._alpha)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>       x = self.norm(x)
>     File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>       result = (val-vmin) / (vmax-vmin)
>     File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>       return divide(self, other)
>     File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>       m |= filled(domain(da, db), True)
>     File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>       return umath.absolute(a) * self.tolerance>= umath.absolute(b)
> MemoryError
>
>
> Thanks for any help,
> -robert
>


These are previous discussions on the issue:

<http://www.mail-archive.com/mat...@li.../msg14727.html>
<http://www.mail-archive.com/mat...@li.../msg19815.html>
<http://www.mail-archive.com/mat...@li.../msg19614.html>

Christoph

Re: [Matplotlib-users] Trouble with imshow

[gary r. <gar...@gm...>]

You might want to try out the visvis module instead of matplotlib for
interactive viewing of large 2D images - my system is also Win64 with
4GB and visvis.imshow() handles a 4k*4k image. You'll probably also
want to disable ipython's object caching if you're doing a lot of this
interactive viewing of large images.

Gary R

On Thu, Feb 3, 2011 at 10:59 AM, Christoph Gohlke <cg...@uc...> wrote:
> On 2/2/2011 3:33 PM, Robert Abiad wrote:
>> Hello All,
>>
>> I'm very new to python, so bear with me.
>>
>> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
>> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
>> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
>> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
>> Here's a transcript.
>>
>>     Welcome to pylab, a matplotlib-based Python environment.
>>     For more information, type 'help(pylab)'.
>>
>> In [1]: import pyfits
>>
>> In [2]: from Tkinter import *
>>
>> In [3]: import tkFileDialog
>>
>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>>
>> In [5]: imshow(image)
>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>>
>> In [6]: close()
>>
>> In [7]: imshow(image,origin='lower')
>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>>
>> In [8]: close()
>>
>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>>
>> In [10]: Exception in Tkinter callback
>> Traceback (most recent call last):
>>     File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>>       return self.func(*args)
>>     File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>>       func(*args)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
>> idle_draw
>>       self.draw()
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>>       FigureCanvasAgg.draw(self)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>>       self.figure.draw(self.renderer)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>       draw(artist, renderer, *args, **kwargs)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>>       func(*args)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>       draw(artist, renderer, *args, **kwargs)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>>       a.draw(renderer)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>       draw(artist, renderer, *args, **kwargs)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>>       im = self.make_image(renderer.get_image_magnification())
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>>       transformed_viewLim)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>>       x = self.to_rgba(self._A, self._alpha)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>>       x = self.norm(x)
>>     File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>>       result = (val-vmin) / (vmax-vmin)
>>     File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>>       return divide(self, other)
>>     File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>>       m |= filled(domain(da, db), True)
>>     File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>>       return umath.absolute(a) * self.tolerance>= umath.absolute(b)
>> MemoryError
>>
>>
>> Thanks for any help,
>> -robert
>>
>
>
> These are previous discussions on the issue:
>
> <http://www.mail-archive.com/mat...@li.../msg14727.html>
> <http://www.mail-archive.com/mat...@li.../msg19815.html>
> <http://www.mail-archive.com/mat...@li.../msg19614.html>
>
> Christoph
>
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>

Re: [Matplotlib-users] Trouble with imshow

[Robert A. <ab...@ss...>]

On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
> On 2/2/2011 3:33 PM, Robert Abiad wrote:
>> Hello All,
>>
>> I'm very new to python, so bear with me.
>>
>> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
>> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
>> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
>> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
>> Here's a transcript.
>>
>>      Welcome to pylab, a matplotlib-based Python environment.
>>      For more information, type 'help(pylab)'.
>>
>> In [1]: import pyfits
>>
>> In [2]: from Tkinter import *
>>
>> In [3]: import tkFileDialog
>>
>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>>
>> In [5]: imshow(image)
>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>>
>> In [6]: close()
>>
>> In [7]: imshow(image,origin='lower')
>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>>
>> In [8]: close()
>>
>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>>
>> In [10]: Exception in Tkinter callback
>> Traceback (most recent call last):
>>      File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>>        return self.func(*args)
>>      File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>>        func(*args)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
>> idle_draw
>>        self.draw()
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>>        FigureCanvasAgg.draw(self)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>>        self.figure.draw(self.renderer)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>        draw(artist, renderer, *args, **kwargs)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>>        func(*args)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>        draw(artist, renderer, *args, **kwargs)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>>        a.draw(renderer)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>        draw(artist, renderer, *args, **kwargs)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>>        im = self.make_image(renderer.get_image_magnification())
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>>        transformed_viewLim)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>>        x = self.to_rgba(self._A, self._alpha)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>>        x = self.norm(x)
>>      File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>>        result = (val-vmin) / (vmax-vmin)
>>      File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>>        return divide(self, other)
>>      File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>>        m |= filled(domain(da, db), True)
>>      File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>>        return umath.absolute(a) * self.tolerance>= umath.absolute(b)
>> MemoryError
>>
>>
>> Thanks for any help,
>> -robert
>>
>
>
> These are previous discussions on the issue:
>
> <http://www.mail-archive.com/mat...@li.../msg14727.html>
> <http://www.mail-archive.com/mat...@li.../msg19815.html>
> <http://www.mail-archive.com/mat...@li.../msg19614.html>
>
> Christoph
>

The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that 
doesn't seem like a great solution.  I use other image processing software (ImageJ and IDL) and 
neither has any trouble with my images.  As I mentioned ImageJ uses 1/10th the memory for the same 
display, or about 1 byte of memory for 1 byte of image.  I think matplotlib should aim for the same. 
  I also think it should free up memory when the image is closed, but maybe I'm not doing the right 
thing.  Is there something else I should be doing to free up memory?

Things are even worse with plot.

I'll file a bug report as Benjamin suggests.

-robert

Re: [Matplotlib-users] Trouble with imshow

[Christoph G. <cg...@uc...>]

On 2/2/2011 5:08 PM, Robert Abiad wrote:
> On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
>> On 2/2/2011 3:33 PM, Robert Abiad wrote:
>>> Hello All,
>>>
>>> I'm very new to python, so bear with me.
>>>
>>> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
>>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
>>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
>>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
>>> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
>>> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
>>> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
>>> Here's a transcript.
>>>
>>>       Welcome to pylab, a matplotlib-based Python environment.
>>>       For more information, type 'help(pylab)'.
>>>
>>> In [1]: import pyfits
>>>
>>> In [2]: from Tkinter import *
>>>
>>> In [3]: import tkFileDialog
>>>
>>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>>>
>>> In [5]: imshow(image)
>>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>>>
>>> In [6]: close()
>>>
>>> In [7]: imshow(image,origin='lower')
>>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>>>
>>> In [8]: close()
>>>
>>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>>>
>>> In [10]: Exception in Tkinter callback
>>> Traceback (most recent call last):
>>>       File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>>>         return self.func(*args)
>>>       File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>>>         func(*args)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
>>> idle_draw
>>>         self.draw()
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>>>         FigureCanvasAgg.draw(self)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>>>         self.figure.draw(self.renderer)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>         draw(artist, renderer, *args, **kwargs)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>>>         func(*args)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>         draw(artist, renderer, *args, **kwargs)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>>>         a.draw(renderer)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>         draw(artist, renderer, *args, **kwargs)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>>>         im = self.make_image(renderer.get_image_magnification())
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>>>         transformed_viewLim)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>>>         x = self.to_rgba(self._A, self._alpha)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>>>         x = self.norm(x)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>>>         result = (val-vmin) / (vmax-vmin)
>>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>>>         return divide(self, other)
>>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>>>         m |= filled(domain(da, db), True)
>>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>>>         return umath.absolute(a) * self.tolerance>= umath.absolute(b)
>>> MemoryError
>>>
>>>
>>> Thanks for any help,
>>> -robert
>>>
>>
>>
>> These are previous discussions on the issue:
>>
>> <http://www.mail-archive.com/mat...@li.../msg14727.html>
>> <http://www.mail-archive.com/mat...@li.../msg19815.html>
>> <http://www.mail-archive.com/mat...@li.../msg19614.html>
>>
>> Christoph
>>
>
> The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that
> doesn't seem like a great solution.  I use other image processing software (ImageJ and IDL) and
> neither has any trouble with my images.  As I mentioned ImageJ uses 1/10th the memory for the same
> display, or about 1 byte of memory for 1 byte of image.  I think matplotlib should aim for the same.
>    I also think it should free up memory when the image is closed, but maybe I'm not doing the right
> thing.  Is there something else I should be doing to free up memory?
>
> Things are even worse with plot.
>
> I'll file a bug report as Benjamin suggests.
>
> -robert
>


Matplotlib converts all images to float64 RGBA before plotting. The 
memory usage increases 32 times when plotting 8 bit images. E.g. at 
least 512 MB contiguous free memory is required for plotting 16 MB uint8 
data.

Christoph

Re: [Matplotlib-users] Trouble with imshow

[Robert A. <ab...@ss...>]

On 2/2/2011 6:06 PM, Eric Firing wrote:
> On 02/02/2011 03:08 PM, Robert Abiad wrote:
>> On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
>>> On 2/2/2011 3:33 PM, Robert Abiad wrote:
>>>> Hello All,
>>>>
>>>> I'm very new to python, so bear with me.
>>>>
>>>> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
>>>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
>>>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
>>>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
>>>> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
>>>> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
>>>> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
>>>> Here's a transcript.
>>>>
>>>>        Welcome to pylab, a matplotlib-based Python environment.
>>>>        For more information, type 'help(pylab)'.
>>>>
>>>> In [1]: import pyfits
>>>>
>>>> In [2]: from Tkinter import *
>>>>
>>>> In [3]: import tkFileDialog
>>>>
>>>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>>>>
>>>> In [5]: imshow(image)
>>>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>>>>
>>>> In [6]: close()
>>>>
>>>> In [7]: imshow(image,origin='lower')
>>>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>>>>
>>>> In [8]: close()
>>>>
>>>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>>>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>>>>
>>>> In [10]: Exception in Tkinter callback
>>>> Traceback (most recent call last):
>>>>        File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>>>>          return self.func(*args)
>>>>        File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>>>>          func(*args)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
>>>> idle_draw
>>>>          self.draw()
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>>>>          FigureCanvasAgg.draw(self)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>>>>          self.figure.draw(self.renderer)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>          draw(artist, renderer, *args, **kwargs)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>>>>          func(*args)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>          draw(artist, renderer, *args, **kwargs)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>>>>          a.draw(renderer)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>          draw(artist, renderer, *args, **kwargs)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>>>>          im = self.make_image(renderer.get_image_magnification())
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>>>>          transformed_viewLim)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>>>>          x = self.to_rgba(self._A, self._alpha)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>>>>          x = self.norm(x)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>>>>          result = (val-vmin) / (vmax-vmin)
>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>>>>          return divide(self, other)
>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>>>>          m |= filled(domain(da, db), True)
>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>>>>          return umath.absolute(a) * self.tolerance>= umath.absolute(b)
>>>> MemoryError
>>>>
>>>>
>>>> Thanks for any help,
>>>> -robert
>>>>
>>>
>>> These are previous discussions on the issue:
>>>
>>> <http://www.mail-archive.com/mat...@li.../msg14727.html>
>>> <http://www.mail-archive.com/mat...@li.../msg19815.html>
>>> <http://www.mail-archive.com/mat...@li.../msg19614.html>
>>>
>>> Christoph
>>>
>> The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that
>> doesn't seem like a great solution.  I use other image processing software (ImageJ and IDL) and
>> neither has any trouble with my images.  As I mentioned ImageJ uses 1/10th the memory for the same
>> display, or about 1 byte of memory for 1 byte of image.  I think matplotlib should aim for the same.
>>     I also think it should free up memory when the image is closed, but maybe I'm not doing the right
>> thing.  Is there something else I should be doing to free up memory?
>>
>> Things are even worse with plot.
>>
>> I'll file a bug report as Benjamin suggests.
> Please file it in the "enhancement" category, not as a "bug".  The
> difficulty that mpl has with images is the result of a design decision
> long ago; as Christoph notes, mpl works primarily with float64 rgba,
> which is horribly memory inefficient, but is very general and works fine
> so long as the image is not too large.  uint8 is already used in some
> circumstances.  I don't know how hard it would be to simply switch to
> uint8 for images, or to make that an option, perhaps for all color
> specification.  It may involve getting into arcana of the extension code
> and the agg library, which is a daunting prospect for most of us,
> certainly for me.  I agree entirely that a major reduction in memory
> usage would be good; patches, or a mpl branch to achieve that, are welcome.
>
> Eric
I'll put it in as an enhancement, but I'm still unsure if there is a bug in 
there as well.  Is there something I should be doing to clear memory after the 
first figure is closed other than close()?  I don't understand why memory usage 
grows each time I replot, but I'm pretty sure it isn't desireable behavior.  As 
I mentioned, this effect is worse with plot.

So is this a bug or improper usage?

As for how to implement a fix for memory usage, I'll let you folks figure that 
out.  But it seems that if I want a grayscale image, I could reduce memory usage 
by 4 if matplotlib could turn rgba into intensity.

-robert

Re: [Matplotlib-users] Trouble with imshow

[Eric F. <ef...@ha...>]

On 02/02/2011 08:38 PM, Robert Abiad wrote:
>
[...]
> I'll put it in as an enhancement, but I'm still unsure if there is a bug in
> there as well.  Is there something I should be doing to clear memory after the
> first figure is closed other than close()?  I don't understand why memory usage
> grows each time I replot, but I'm pretty sure it isn't desireable behavior.  As
> I mentioned, this effect is worse with plot.
>
> So is this a bug or improper usage?

I'm not quite sure, but I don't think there is a specifically matplotlib 
memory leak bug at work here.  Are you using ipython, and if so, have 
you turned off the caching?  In its default mode, ipython keeps lots of 
references, thereby keeping memory in use.  Also, memory management and 
reporting can be a bit tricky and misleading.

Nevertheless, the attached script may be illustrating the problem.  Try 
running it from the command line as-is (maybe shorten the loop--it 
doesn't take 100 iterations to show the pattern) and then commenting out 
the line as indicated in the comment.  It seems that if anything is done 
that adds ever so slightly to memory use while the figure is displayed, 
then when the figure is closed, its memory is not reused.  I'm puzzled.

Eric

>
> As for how to implement a fix for memory usage, I'll let you folks figure that
> out.  But it seems that if I want a grayscale image, I could reduce memory usage
> by 4 if matplotlib could turn rgba into intensity.
>
> -robert

Re: [Matplotlib-users] Trouble with imshow

[Christoph G. <cg...@uc...>]

On 2/3/2011 3:13 PM, Eric Firing wrote:
> On 02/03/2011 01:02 PM, Christoph Gohlke wrote:
>>
>>
>> On 2/3/2011 2:44 PM, Eric Firing wrote:
>>> On 02/03/2011 12:28 PM, Christoph Gohlke wrote:
>>>>
>>>>
>>>> On 2/3/2011 2:15 PM, Eric Firing wrote:
>>>>> On 02/03/2011 11:30 AM, Robert Abiad wrote:
>>>>>> On 2/3/2011 10:06 AM, Eric Firing wrote:
>>>>>>> On 02/02/2011 10:17 PM, Eric Firing wrote:
>>>>>>>> On 02/02/2011 08:38 PM, Robert Abiad wrote:
>>>>>>>>>
>>>>>>>> [...]
>>>>>>>>> I'll put it in as an enhancement, but I'm still unsure if there is a
>>>>>>>>> bug in
>>>>>>>>> there as well. Is there something I should be doing to clear memory
>>>>>>>>> after the
>>>>>>>>> first figure is closed other than close()? I don't understand why
>>>>>>>>> memory usage
>>>>>>>>> grows each time I replot, but I'm pretty sure it isn't desireable
>>>>>>>>> behavior. As
>>>>>>>>> I mentioned, this effect is worse with plot.
>>>>>>>>>
>>>>>>>>> So is this a bug or improper usage?
>>>>>>>>
>>>>>>>> I'm not quite sure, but I don't think there is a specifically
>>>>>>>> matplotlib
>>>>>>>> memory leak bug at work here. Are you using ipython, and if so, have
>>>>>>>> you
>>>>>>>> turned off the caching? In its default mode, ipython keeps lots of
>>>>>>>> references, thereby keeping memory in use. Also, memory management and
>>>>>>>> reporting can be a bit tricky and misleading.
>>>>>>>>
>>>>>>>> Nevertheless, the attached script may be illustrating the problem. Try
>>>>>>>> running it from the command line as-is (maybe shorten the loop--it
>>>>>>>> doesn't take 100 iterations to show the pattern) and then commenting
>>>>>>>> out
>>>>>>>> the line as indicated in the comment. It seems that if anything is done
>>>>>>>> that adds ever so slightly to memory use while the figure is displayed,
>>>>>>>> then when the figure is closed, its memory is not reused. I'm puzzled.
>>>>>>>
>>>>>>> I wasn't thinking straight--there is no mystery and no memory leak.
>>>>>>> Ignore my example script referred to above. It was saving rows of the z
>>>>>>> array, not single elements as I had intended, so of course memory use
>>>>>>> was growing substantially.
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>
>>>>>> You may not see a memory leak, but I still can't get my memory back
>>>>>> without killing python. I
>>>>>> turned off the ipython caching and even ran without iPython on both
>>>>>> Windows and Ubuntu, but when I
>>>>>> use imshow(), followed by close('all') and another imshow(), I run out
>>>>>> of memory. I can see from
>>>>>> the OS that the memory does not come back after close() and that it
>>>>>> grows after the second imshow().
>>>>>>
>>>>>> Any other ideas? Looks like a bug to me otherwise.
>>>>>
>>>>> Except that I tried the same things and did not get quite the same
>>>>> result. Let's track this down. Please try the attached script, and see
>>>>> if the memory usage grows substantially, or just oscillates a bit.
>>>>>
>>>>> Eric
>>>>>
>>>>
>>>>
>>>> One thing I noticed is that if I add a "def __del__(self): print 'del'"
>>>> to image._AxesImageBase, it never gets called. _AxesImageBase keeps
>>>> float64 and uint8 rgba images in a cache, which is never freed.
>>>
>>> Adding a __del__ method defeats (or blocks) the garbage collection.
>>>
>>
>> Sorry, never heard of that. I thought __del__() is called when the
>> reference count reaches 0.
>
> It is, but if there are circular reference chains (cycles--and mpl is
> full of them) then the garbage collector has to identify them and remove
> them.  If it encounters a __del__ it stops and leaves that cycle alone.
>

My understanding is that if there is a circular reference then the 
refcount will not be zero anyway. In this case _AxesImageBase instances 
and their image caches will never be deleted by the gc (__del__ method 
present or not) unless the circle is broken. When the interpreter quits 
things are deleted by other means. I don't know matplotlib code good 
enough to fix this and will instead work on reducing the memory overhead 
needed to plot an image. In the meantime it could help to explicitly 
delete the image cache when a plot closes or to avoid caching altogether.

Christoph


> Eric
>
>>
>> Christoph
>>
>>> Since self._imcache is an instance attribute, when the instance is no
>>> longer referenced, it should get garbage-collected, provided there is no
>>> __del__ method.
>>>
>>> Eric
>>>
>>>>
>>>> Christoph
>
> ------------------------------------------------------------------------------
> The modern datacenter depends on network connectivity to access resources
> and provide services. The best practices for maximizing a physical server's
> connectivity to a physical network are well understood - see how these
> rules translate into the virtual world?
> http://p.sf.net/sfu/oracle-sfdevnlfb
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>

Re: [Matplotlib-users] Trouble with imshow

[Benjamin R. <ben...@ou...>]

On Thu, Feb 3, 2011 at 6:37 PM, Christoph Gohlke <cg...@uc...> wrote:

>
>
> On 2/3/2011 3:13 PM, Eric Firing wrote:
> > On 02/03/2011 01:02 PM, Christoph Gohlke wrote:
> >>
> >>
> >> On 2/3/2011 2:44 PM, Eric Firing wrote:
> >>> On 02/03/2011 12:28 PM, Christoph Gohlke wrote:
> >>>>
> >>>>
> >>>> On 2/3/2011 2:15 PM, Eric Firing wrote:
> >>>>> On 02/03/2011 11:30 AM, Robert Abiad wrote:
> >>>>>> On 2/3/2011 10:06 AM, Eric Firing wrote:
> >>>>>>> On 02/02/2011 10:17 PM, Eric Firing wrote:
> >>>>>>>> On 02/02/2011 08:38 PM, Robert Abiad wrote:
> >>>>>>>>>
> >>>>>>>> [...]
> >>>>>>>>> I'll put it in as an enhancement, but I'm still unsure if there
> is a
> >>>>>>>>> bug in
> >>>>>>>>> there as well. Is there something I should be doing to clear
> memory
> >>>>>>>>> after the
> >>>>>>>>> first figure is closed other than close()? I don't understand why
> >>>>>>>>> memory usage
> >>>>>>>>> grows each time I replot, but I'm pretty sure it isn't desireable
> >>>>>>>>> behavior. As
> >>>>>>>>> I mentioned, this effect is worse with plot.
> >>>>>>>>>
> >>>>>>>>> So is this a bug or improper usage?
> >>>>>>>>
> >>>>>>>> I'm not quite sure, but I don't think there is a specifically
> >>>>>>>> matplotlib
> >>>>>>>> memory leak bug at work here. Are you using ipython, and if so,
> have
> >>>>>>>> you
> >>>>>>>> turned off the caching? In its default mode, ipython keeps lots of
> >>>>>>>> references, thereby keeping memory in use. Also, memory management
> and
> >>>>>>>> reporting can be a bit tricky and misleading.
> >>>>>>>>
> >>>>>>>> Nevertheless, the attached script may be illustrating the problem.
> Try
> >>>>>>>> running it from the command line as-is (maybe shorten the loop--it
> >>>>>>>> doesn't take 100 iterations to show the pattern) and then
> commenting
> >>>>>>>> out
> >>>>>>>> the line as indicated in the comment. It seems that if anything is
> done
> >>>>>>>> that adds ever so slightly to memory use while the figure is
> displayed,
> >>>>>>>> then when the figure is closed, its memory is not reused. I'm
> puzzled.
> >>>>>>>
> >>>>>>> I wasn't thinking straight--there is no mystery and no memory leak.
> >>>>>>> Ignore my example script referred to above. It was saving rows of
> the z
> >>>>>>> array, not single elements as I had intended, so of course memory
> use
> >>>>>>> was growing substantially.
> >>>>>>>
> >>>>>>> Eric
> >>>>>>>
> >>>>>>
> >>>>>> You may not see a memory leak, but I still can't get my memory back
> >>>>>> without killing python. I
> >>>>>> turned off the ipython caching and even ran without iPython on both
> >>>>>> Windows and Ubuntu, but when I
> >>>>>> use imshow(), followed by close('all') and another imshow(), I run
> out
> >>>>>> of memory. I can see from
> >>>>>> the OS that the memory does not come back after close() and that it
> >>>>>> grows after the second imshow().
> >>>>>>
> >>>>>> Any other ideas? Looks like a bug to me otherwise.
> >>>>>
> >>>>> Except that I tried the same things and did not get quite the same
> >>>>> result. Let's track this down. Please try the attached script, and
> see
> >>>>> if the memory usage grows substantially, or just oscillates a bit.
> >>>>>
> >>>>> Eric
> >>>>>
> >>>>
> >>>>
> >>>> One thing I noticed is that if I add a "def __del__(self): print
> 'del'"
> >>>> to image._AxesImageBase, it never gets called. _AxesImageBase keeps
> >>>> float64 and uint8 rgba images in a cache, which is never freed.
> >>>
> >>> Adding a __del__ method defeats (or blocks) the garbage collection.
> >>>
> >>
> >> Sorry, never heard of that. I thought __del__() is called when the
> >> reference count reaches 0.
> >
> > It is, but if there are circular reference chains (cycles--and mpl is
> > full of them) then the garbage collector has to identify them and remove
> > them.  If it encounters a __del__ it stops and leaves that cycle alone.
> >
>
> My understanding is that if there is a circular reference then the
> refcount will not be zero anyway. In this case _AxesImageBase instances
> and their image caches will never be deleted by the gc (__del__ method
> present or not) unless the circle is broken. When the interpreter quits
> things are deleted by other means. I don't know matplotlib code good
> enough to fix this and will instead work on reducing the memory overhead
> needed to plot an image. In the meantime it could help to explicitly
> delete the image cache when a plot closes or to avoid caching altogether.
>
> Christoph
>
>
Just to clarify the circular reference paradox.  The garbage collector will
pick it up because it is using weak refs instead of regular references.  So,
when a circular reference occurs, but is not hard ref-ed to anything, then
it is "detached" and the gc picks it up.  I am not very familiar on this
concept in particular, but I do remember it being explained this way.

Anyone who is more knowledgeable about this, I would welcome further comment
or corrections to what I remember.

Also, not to sound too annoying, but has anyone considered the idea of using
compressed arrays for holding those rgba values?

Ben Root

Re: [Matplotlib-users] Trouble with imshow

[Pauli V. <pa...@ik...>]

Thu, 03 Feb 2011 16:37:31 -0800, Christoph Gohlke wrote:
[clip]
> My understanding is that if there is a circular reference then the
> refcount will not be zero anyway. In this case _AxesImageBase instances
> and their image caches will never be deleted by the gc (__del__ method
> present or not) unless the circle is broken. 

Python's GC will try to break the cycles if possible. In fact, there's no 
other need for a GC in a reference counted system.

http://docs.python.org/extending/extending.html#reference-counts

-- 
Pauli Virtanen

Re: [Matplotlib-users] Trouble with imshow

[Eric F. <ef...@ha...>]

On 02/04/2011 10:28 AM, Christoph Gohlke wrote:
>
>
> On 2/4/2011 11:54 AM, Eric Firing wrote:
>> On 02/03/2011 05:35 PM, Christoph Gohlke wrote:
>>>
>>>
>>> On 2/3/2011 6:50 PM, Eric Firing wrote:
>>>> On 02/03/2011 03:04 PM, Benjamin Root wrote:
>>>>
>>>>> Also, not to sound too annoying, but has anyone considered the idea of
>>>>> using compressed arrays for holding those rgba values?
>>>>
>>>> I don't see how that really helps; as far as I know, a full rgba array
>>>> has to be passed into agg. What *does* help is using uint8 from start
>>>> to finish. It might also be possible to use some smart downsampling
>>>> before generating the rgba array, but the uint8 route seems to me the
>>>> first thing to attack.
>>>>
>>>> Eric
>>>>
>>>>>
>>>>> Ben Root
>>>>
>>>
>>> Please review the attached patch. It avoids generating and storing
>>> float64 rgba arrays and uses uint8 rgba instead. That's a huge memory
>>> saving and also faster. I can't see any side effects as
>>> _image.fromarray() converts the float64 input to uint8 anyway.
>>
>> Christoph,
>>
>> Thank you!  I haven't found anything wrong with that delightfully simple
>> patch, so I have committed it to the trunk.  Back in 2007 I added the
>> ability of colormapping to generate uint8 directly, precisely to enable
>> this sort of optimization.  Why it was not already being used in imshow,
>> I don't know--maybe I was going to do it, got sidetracked, and never
>> finished.
>>
>> I suspect it won't be as simple as for the plain image, but there may be
>> opportunities for optimizing with uint8 in other image-like operations.
>>
>>>
>>> So far other attempts to optimize memory usage were thwarted by
>>> matplotlib's internal use of masked arrays. As mentioned before, users
>>> can provide their own normalized rgba arrays to avoid all this processing.
>>>
>>
>> Did you see other potential low-hanging fruit that might be harvested
>> with some changes to the code associated with masked arrays?
>>
>> Eric
>>
>
> The norm function currently converts the data to double precision
> floating point and also creates temporary arrays that can be avoided.
> For float32 and low precision integer images this seems overkill and one
> could use float32. It might be possible to replace the norm function
> with numpy.digitize if that works with masked arrays. Last, the
> _image.frombyte function does a copy of 'strided arrays' (only relevant
> when zooming/panning large images). I try to provide a patch for each.

masked arrays can be filled to create an ndarray before passing to 
digitize; whether that will be faster, remains to be seen.  I've never 
used digitize.

Regarding frombyte, I suspect you can't avoid the copy; the data 
structure being passed to agg is just a string of bytes, as far as I can 
see, so everything is based on having a simple contiguous array.

Eric

>
> Christoph
>
> ------------------------------------------------------------------------------
> The modern datacenter depends on network connectivity to access resources
> and provide services. The best practices for maximizing a physical server's
> connectivity to a physical network are well understood - see how these
> rules translate into the virtual world?
> http://p.sf.net/sfu/oracle-sfdevnlfb
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Re: [Matplotlib-users] Trouble with imshow

[Eric F. <ef...@ha...>]

On 02/08/2011 02:39 PM, Christoph Gohlke wrote:
>

>
> Please consider the attached patch for the _image.frombyte function. It
> avoids temporary copies in case of non-contiguous input arrays. Copying
> a 1024x1024 slice out of a contiguous 4096x4096 RGBA or RGB array is
> about 7x faster (a common case for zooming/panning). Copying contiguous
> RGB input arrays is ~2x faster. Tested on win32-py2.7.
>
> Christoph
>

Thank you!

Looks good, speeds up zooming and panning on large images as advertised. 
  An 8000x8000 image is actually manageable now. 
interpolation='nearest' is still very slow until the image is 
substantially zoomed, but everything is quite quick with other 
interpolation styles.  The slowness of 'nearest' looks like a basic 
characteristic of the implementation.

I committed the patch in 8966.

Before that I found and committed a big speed-up in Normalize.

Eric

Re: [Matplotlib-users] Trouble with imshow

[Benjamin R. <ben...@ou...>]

On Wed, Feb 9, 2011 at 1:50 AM, Eric Firing <ef...@ha...> wrote:

> On 02/08/2011 02:39 PM, Christoph Gohlke wrote:
> >
>
> >
> > Please consider the attached patch for the _image.frombyte function. It
> > avoids temporary copies in case of non-contiguous input arrays. Copying
> > a 1024x1024 slice out of a contiguous 4096x4096 RGBA or RGB array is
> > about 7x faster (a common case for zooming/panning). Copying contiguous
> > RGB input arrays is ~2x faster. Tested on win32-py2.7.
> >
> > Christoph
> >
>
> Thank you!
>
> Looks good, speeds up zooming and panning on large images as advertised.
>  An 8000x8000 image is actually manageable now.
> interpolation='nearest' is still very slow until the image is
> substantially zoomed, but everything is quite quick with other
> interpolation styles.  The slowness of 'nearest' looks like a basic
> characteristic of the implementation.
>
> I committed the patch in 8966.
>
> Before that I found and committed a big speed-up in Normalize.
>
> Eric
>
>
Eric,

How much is the speed-up in Normalize?  It might be worth it to apply it to
LogNorm as well.  (as an aside, I find the duplication of code in Normalize
and friends a little disconcerting...).

Also, what would it take (if it is at all possible) to take advantage of
these image optimizations while using pcolor?

Ben Root

Re: [Matplotlib-users] Trouble with imshow

[Eric F. <ef...@ha...>]

On 02/09/2011 12:36 PM, Benjamin Root wrote:
> On Wed, Feb 9, 2011 at 1:50 AM, Eric Firing <ef...@ha...
> <mailto:ef...@ha...>> wrote:
>
>     On 02/08/2011 02:39 PM, Christoph Gohlke wrote:
>      >
>
>      >
>      > Please consider the attached patch for the _image.frombyte
>     function. It
>      > avoids temporary copies in case of non-contiguous input arrays.
>     Copying
>      > a 1024x1024 slice out of a contiguous 4096x4096 RGBA or RGB array is
>      > about 7x faster (a common case for zooming/panning). Copying
>     contiguous
>      > RGB input arrays is ~2x faster. Tested on win32-py2.7.
>      >
>      > Christoph
>      >
>
>     Thank you!
>
>     Looks good, speeds up zooming and panning on large images as advertised.
>       An 8000x8000 image is actually manageable now.
>     interpolation='nearest' is still very slow until the image is
>     substantially zoomed, but everything is quite quick with other
>     interpolation styles.  The slowness of 'nearest' looks like a basic
>     characteristic of the implementation.
>
>     I committed the patch in 8966.
>
>     Before that I found and committed a big speed-up in Normalize.
>
>     Eric
>
>
> Eric,
>
> How much is the speed-up in Normalize?  It might be worth it to apply it
> to LogNorm as well.  (as an aside, I find the duplication of code in
> Normalize and friends a little disconcerting...).

Timing: I don't recall exactly, but it was along the lines of 3.5 
seconds getting knocked down to 0.6 seconds.

Yes, I will apply it to LogNorm.

Duplication: yes, but every time I look at it I see that there are 
subtle but essential differences here and there, so it does not look 
worthwhile to try to factor out the remaining commonality.

>
> Also, what would it take (if it is at all possible) to take advantage of
> these image optimizations while using pcolor?

The norm and cmap optimizations are already taken advantage of by all 
color mapping operations.  The optimization of using uint8 was already 
being done in quadmesh, nonuniform image, and pcolor image (all used by 
pcolorfast).  The image.frombyte optimization that Christoph just came 
up with (8966) is specific to image.  There may be some similar 
optimizations that could be made for other image-like plots (reducing 
copying and temporaries), but I don't think there will be the same level 
of payoff.  Pcolor itself is hopelessly slow by nature, and should be 
used only for relatively small arrays.  And then there is hexbin, which 
is *really* slow.  It could be sped up with the addition of some cython, 
but would need a quadmesh-like extension to make it really speedy.  (All 
this is off the top of my head, so I may be misstating a point here and 
there.)

Eric

>
> Ben Root
>
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
>
>
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Re: [Matplotlib-users] Trouble with imshow

[Eric F. <ef...@ha...>]

On 02/09/2011 02:29 PM, Benjamin Root wrote:
> On Wed, Feb 9, 2011 at 1:50 AM, Eric Firing <ef...@ha...
> <mailto:ef...@ha...>> wrote:
>
>     On 02/08/2011 02:39 PM, Christoph Gohlke wrote:
>      >
>
>      >
>      > Please consider the attached patch for the _image.frombyte
>     function. It
>      > avoids temporary copies in case of non-contiguous input arrays.
>     Copying
>      > a 1024x1024 slice out of a contiguous 4096x4096 RGBA or RGB array is
>      > about 7x faster (a common case for zooming/panning). Copying
>     contiguous
>      > RGB input arrays is ~2x faster. Tested on win32-py2.7.
>      >
>      > Christoph
>      >
>
>     Thank you!
>
>     Looks good, speeds up zooming and panning on large images as advertised.
>       An 8000x8000 image is actually manageable now.
>     interpolation='nearest' is still very slow until the image is
>     substantially zoomed, but everything is quite quick with other
>     interpolation styles.  The slowness of 'nearest' looks like a basic
>     characteristic of the implementation.
>
>     I committed the patch in 8966.
>
>     Before that I found and committed a big speed-up in Normalize.
>
>     Eric
>
>
> Bug Report:
>
> At some point between the recent revision and r8934, setting the alpha
> value to anythhing but None will cause the image to not show.  I suspect
> it has something to do with some of the recent revisions.  Maybe the
> alpha values were being converted into an integer, causing them to be
> zero?  Then again, even setting alpha to 1 will cause the image to
> disappear.
>
> Ideas?  Thoughts?  I included an example script below.

Thanks for the report.  I'll fix it some time today.

Eric


>
> Ben Root
>
>
> Example script:
>
>
> import numpy as np
> import matplotlib.pyplot as plt
>
> z = np.random.random((40, 50))
>
> fig = plt.figure()
> ax = fig.add_subplot(1, 2, 1)
> ax.imshow(z, alpha=1.0)
> ax.set_title('Blank!')
>
> ax = fig.add_subplot(1, 2, 2)
> ax.imshow(z, alpha=None)
> ax.set_title("Not Blank")
>
>
>
> plt.show()
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
>
>
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Re: [Matplotlib-users] Trouble with imshow

[Eric F. <ef...@ha...>]

On 02/09/2011 02:29 PM, Benjamin Root wrote:

>
> Bug Report:
>
> At some point between the recent revision and r8934, setting the alpha
> value to anythhing but None will cause the image to not show.  I suspect
> it has something to do with some of the recent revisions.  Maybe the
> alpha values were being converted into an integer, causing them to be
> zero?  Then again, even setting alpha to 1 will cause the image to
> disappear.
>
> Ideas?  Thoughts?  I included an example script below.

Fixed in 8968.

Re: [Matplotlib-users] Trouble with imshow

[Benjamin R. <ben...@ou...>]

On Wed, Feb 9, 2011 at 7:22 PM, Eric Firing <ef...@ha...> wrote:

> On 02/09/2011 02:29 PM, Benjamin Root wrote:
>
> >
> > Bug Report:
> >
> > At some point between the recent revision and r8934, setting the alpha
> > value to anythhing but None will cause the image to not show.  I suspect
> > it has something to do with some of the recent revisions.  Maybe the
> > alpha values were being converted into an integer, causing them to be
> > zero?  Then again, even setting alpha to 1 will cause the image to
> > disappear.
> >
> > Ideas?  Thoughts?  I included an example script below.
>
> Fixed in 8968.
>
>
Thanks!

Re: [Matplotlib-users] Trouble with imshow

[Eric F. <ef...@ha...>]

On 02/02/2011 03:08 PM, Robert Abiad wrote:
> On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
>> On 2/2/2011 3:33 PM, Robert Abiad wrote:
>>> Hello All,
>>>
>>> I'm very new to python, so bear with me.
>>>
>>> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
>>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
>>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
>>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
>>> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
>>> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
>>> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
>>> Here's a transcript.
>>>
>>>       Welcome to pylab, a matplotlib-based Python environment.
>>>       For more information, type 'help(pylab)'.
>>>
>>> In [1]: import pyfits
>>>
>>> In [2]: from Tkinter import *
>>>
>>> In [3]: import tkFileDialog
>>>
>>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>>>
>>> In [5]: imshow(image)
>>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>>>
>>> In [6]: close()
>>>
>>> In [7]: imshow(image,origin='lower')
>>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>>>
>>> In [8]: close()
>>>
>>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>>>
>>> In [10]: Exception in Tkinter callback
>>> Traceback (most recent call last):
>>>       File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>>>         return self.func(*args)
>>>       File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>>>         func(*args)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
>>> idle_draw
>>>         self.draw()
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>>>         FigureCanvasAgg.draw(self)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>>>         self.figure.draw(self.renderer)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>         draw(artist, renderer, *args, **kwargs)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>>>         func(*args)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>         draw(artist, renderer, *args, **kwargs)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>>>         a.draw(renderer)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>         draw(artist, renderer, *args, **kwargs)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>>>         im = self.make_image(renderer.get_image_magnification())
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>>>         transformed_viewLim)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>>>         x = self.to_rgba(self._A, self._alpha)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>>>         x = self.norm(x)
>>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>>>         result = (val-vmin) / (vmax-vmin)
>>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>>>         return divide(self, other)
>>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>>>         m |= filled(domain(da, db), True)
>>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>>>         return umath.absolute(a) * self.tolerance>= umath.absolute(b)
>>> MemoryError
>>>
>>>
>>> Thanks for any help,
>>> -robert
>>>
>>
>>
>> These are previous discussions on the issue:
>>
>> <http://www.mail-archive.com/mat...@li.../msg14727.html>
>> <http://www.mail-archive.com/mat...@li.../msg19815.html>
>> <http://www.mail-archive.com/mat...@li.../msg19614.html>
>>
>> Christoph
>>
>
> The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that
> doesn't seem like a great solution.  I use other image processing software (ImageJ and IDL) and
> neither has any trouble with my images.  As I mentioned ImageJ uses 1/10th the memory for the same
> display, or about 1 byte of memory for 1 byte of image.  I think matplotlib should aim for the same.
>    I also think it should free up memory when the image is closed, but maybe I'm not doing the right
> thing.  Is there something else I should be doing to free up memory?
>
> Things are even worse with plot.
>
> I'll file a bug report as Benjamin suggests.

Please file it in the "enhancement" category, not as a "bug".  The 
difficulty that mpl has with images is the result of a design decision 
long ago; as Christoph notes, mpl works primarily with float64 rgba, 
which is horribly memory inefficient, but is very general and works fine 
so long as the image is not too large.  uint8 is already used in some 
circumstances.  I don't know how hard it would be to simply switch to 
uint8 for images, or to make that an option, perhaps for all color 
specification.  It may involve getting into arcana of the extension code 
and the agg library, which is a daunting prospect for most of us, 
certainly for me.  I agree entirely that a major reduction in memory 
usage would be good; patches, or a mpl branch to achieve that, are welcome.

Eric

>
> -robert
>
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Re: [Matplotlib-users] Trouble with imshow

[Christoph G. <cg...@uc...>]

On 2/2/2011 6:06 PM, Eric Firing wrote:
> On 02/02/2011 03:08 PM, Robert Abiad wrote:
>> On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
>>> On 2/2/2011 3:33 PM, Robert Abiad wrote:
>>>> Hello All,
>>>>
>>>> I'm very new to python, so bear with me.
>>>>
>>>> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
>>>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
>>>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
>>>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
>>>> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
>>>> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
>>>> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
>>>> Here's a transcript.
>>>>
>>>>        Welcome to pylab, a matplotlib-based Python environment.
>>>>        For more information, type 'help(pylab)'.
>>>>
>>>> In [1]: import pyfits
>>>>
>>>> In [2]: from Tkinter import *
>>>>
>>>> In [3]: import tkFileDialog
>>>>
>>>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>>>>
>>>> In [5]: imshow(image)
>>>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>>>>
>>>> In [6]: close()
>>>>
>>>> In [7]: imshow(image,origin='lower')
>>>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>>>>
>>>> In [8]: close()
>>>>
>>>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>>>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>>>>
>>>> In [10]: Exception in Tkinter callback
>>>> Traceback (most recent call last):
>>>>        File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>>>>          return self.func(*args)
>>>>        File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>>>>          func(*args)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
>>>> idle_draw
>>>>          self.draw()
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>>>>          FigureCanvasAgg.draw(self)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>>>>          self.figure.draw(self.renderer)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>          draw(artist, renderer, *args, **kwargs)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>>>>          func(*args)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>          draw(artist, renderer, *args, **kwargs)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>>>>          a.draw(renderer)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>          draw(artist, renderer, *args, **kwargs)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>>>>          im = self.make_image(renderer.get_image_magnification())
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>>>>          transformed_viewLim)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>>>>          x = self.to_rgba(self._A, self._alpha)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>>>>          x = self.norm(x)
>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>>>>          result = (val-vmin) / (vmax-vmin)
>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>>>>          return divide(self, other)
>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>>>>          m |= filled(domain(da, db), True)
>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>>>>          return umath.absolute(a) * self.tolerance>= umath.absolute(b)
>>>> MemoryError
>>>>
>>>>
>>>> Thanks for any help,
>>>> -robert
>>>>
>>>
>>>
>>> These are previous discussions on the issue:
>>>
>>> <http://www.mail-archive.com/mat...@li.../msg14727.html>
>>> <http://www.mail-archive.com/mat...@li.../msg19815.html>
>>> <http://www.mail-archive.com/mat...@li.../msg19614.html>
>>>
>>> Christoph
>>>
>>
>> The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that
>> doesn't seem like a great solution.  I use other image processing software (ImageJ and IDL) and
>> neither has any trouble with my images.  As I mentioned ImageJ uses 1/10th the memory for the same
>> display, or about 1 byte of memory for 1 byte of image.  I think matplotlib should aim for the same.
>>     I also think it should free up memory when the image is closed, but maybe I'm not doing the right
>> thing.  Is there something else I should be doing to free up memory?
>>
>> Things are even worse with plot.
>>
>> I'll file a bug report as Benjamin suggests.
>
> Please file it in the "enhancement" category, not as a "bug".  The
> difficulty that mpl has with images is the result of a design decision
> long ago; as Christoph notes, mpl works primarily with float64 rgba,
> which is horribly memory inefficient, but is very general and works fine
> so long as the image is not too large.  uint8 is already used in some
> circumstances.  I don't know how hard it would be to simply switch to
> uint8 for images, or to make that an option, perhaps for all color
> specification.  It may involve getting into arcana of the extension code
> and the agg library, which is a daunting prospect for most of us,
> certainly for me.  I agree entirely that a major reduction in memory
> usage would be good; patches, or a mpl branch to achieve that, are welcome.
>
> Eric
>


How about special-casing uint8 arrays, assuming that these don't need to 
be normalized? This gives the user the option to bypass matplotlib's 
internal conversions and saves around 450 MB for plotting one 4096x4096 
uint8 image. A preliminary, mostly untested, patch is attached. Similar 
changes could be done for other signed and unsigned integer data.

Christoph

Re: [Matplotlib-users] Trouble with imshow

[Benjamin R. <ben...@ou...>]

On Wed, Feb 2, 2011 at 8:06 PM, Eric Firing <ef...@ha...> wrote:

> On 02/02/2011 03:08 PM, Robert Abiad wrote:
> > On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
> >> On 2/2/2011 3:33 PM, Robert Abiad wrote:
> >>> Hello All,
> >>>
> >>> I'm very new to python, so bear with me.
> >>>
> >>> I'd like to use python to do my image processing, but I'm running into
> behavior that doesn't make
> >>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory,
> python 2.6.6, and the newest
> >>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.
>  I'm loading in a fits file
> >>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see
> memory usage go up by 50MB,
> >>> but when I try displaying the image using imshow(), my memory usage
> goes up by around 500MB, each
> >>> time.  If I close the figure and replot it, imshow() crashes.  I don't
> know if I'm doing something
> >>> wrong, or if it's a new or known bug.  I tried the same thing on Linux
> and got the same result.
> >>> Here's a transcript.
> >>>
> >>>       Welcome to pylab, a matplotlib-based Python environment.
> >>>       For more information, type 'help(pylab)'.
> >>>
> >>> In [1]: import pyfits
> >>>
> >>> In [2]: from Tkinter import *
> >>>
> >>> In [3]: import tkFileDialog
> >>>
> >>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
> >>>
> >>> In [5]: imshow(image)
> >>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
> >>>
> >>> In [6]: close()
> >>>
> >>> In [7]: imshow(image,origin='lower')
> >>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
> >>>
> >>> In [8]: close()
> >>>
> >>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
> >>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
> >>>
> >>> In [10]: Exception in Tkinter callback
> >>> Traceback (most recent call last):
> >>>       File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in
> __call__
> >>>         return self.func(*args)
> >>>       File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in
> callit
> >>>         func(*args)
> >>>       File
> "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py",
> line 263, in
> >>> idle_draw
> >>>         self.draw()
> >>>       File
> "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py",
> line 248, in draw
> >>>         FigureCanvasAgg.draw(self)
> >>>       File
> "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 394, in draw
> >>>         self.figure.draw(self.renderer)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py",
> line 55, in draw_wrapper
> >>>         draw(artist, renderer, *args, **kwargs)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py",
> line 798, in draw
> >>>         func(*args)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py",
> line 55, in draw_wrapper
> >>>         draw(artist, renderer, *args, **kwargs)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py",
> line 1946, in draw
> >>>         a.draw(renderer)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py",
> line 55, in draw_wrapper
> >>>         draw(artist, renderer, *args, **kwargs)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py",
> line 354, in draw
> >>>         im = self.make_image(renderer.get_image_magnification())
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py",
> line 569, in make_image
> >>>         transformed_viewLim)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py",
> line 201, in _get_unsampled_image
> >>>         x = self.to_rgba(self._A, self._alpha)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line
> 193, in to_rgba
> >>>         x = self.norm(x)
> >>>       File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py",
> line 820, in __call__
> >>>         result = (val-vmin) / (vmax-vmin)
> >>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line
> 3673, in __div__
> >>>         return divide(self, other)
> >>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line
> 1077, in __call__
> >>>         m |= filled(domain(da, db), True)
> >>>       File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line
> 772, in __call__
> >>>         return umath.absolute(a) * self.tolerance>= umath.absolute(b)
> >>> MemoryError
> >>>
> >>>
> >>> Thanks for any help,
> >>> -robert
> >>>
> >>
> >>
> >> These are previous discussions on the issue:
> >>
> >> <
> http://www.mail-archive.com/mat...@li.../msg14727.html
> >
> >> <
> http://www.mail-archive.com/mat...@li.../msg19815.html
> >
> >> <
> http://www.mail-archive.com/mat...@li.../msg19614.html
> >
> >>
> >> Christoph
> >>
> >
> > The first 2 discussions lead to suggestions of more memory on a 64-bit
> installation, but that
> > doesn't seem like a great solution.  I use other image processing
> software (ImageJ and IDL) and
> > neither has any trouble with my images.  As I mentioned ImageJ uses
> 1/10th the memory for the same
> > display, or about 1 byte of memory for 1 byte of image.  I think
> matplotlib should aim for the same.
> >    I also think it should free up memory when the image is closed, but
> maybe I'm not doing the right
> > thing.  Is there something else I should be doing to free up memory?
> >
> > Things are even worse with plot.
> >
> > I'll file a bug report as Benjamin suggests.
>
> Please file it in the "enhancement" category, not as a "bug".  The
> difficulty that mpl has with images is the result of a design decision
> long ago; as Christoph notes, mpl works primarily with float64 rgba,
> which is horribly memory inefficient, but is very general and works fine
> so long as the image is not too large.  uint8 is already used in some
> circumstances.  I don't know how hard it would be to simply switch to
> uint8 for images, or to make that an option, perhaps for all color
> specification.  It may involve getting into arcana of the extension code
> and the agg library, which is a daunting prospect for most of us,
> certainly for me.  I agree entirely that a major reduction in memory
> usage would be good; patches, or a mpl branch to achieve that, are welcome.
>
> Eric
>
>
As a possible stop-gap solution, maybe we could consider the compressed
arrays (carry)?

I am not a fan of doing special cases, though, especially at the very
base/core portions of the library.

Ben Root

Re: [Matplotlib-users] Trouble with imshow

[Christoph G. <cg...@uc...>]

On 2/2/2011 8:22 PM, Benjamin Root wrote:
> On Wed, Feb 2, 2011 at 8:06 PM, Eric Firing <ef...@ha...
> <mailto:ef...@ha...>> wrote:
>
>     On 02/02/2011 03:08 PM, Robert Abiad wrote:
>     >  On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
>     > > On 2/2/2011 3:33 PM, Robert Abiad wrote:
>     > >> Hello All,
>     > >>
>     > >> I'm very new to python, so bear with me.
>     > >>
>     > >> I'd like to use python to do my image processing, but I'm
>     running into behavior that doesn't make
>     > >> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of
>     memory, python 2.6.6, and the newest
>     > >> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1),
>     scipy.  I'm loading in a fits file
>     > >> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I
>     can see memory usage go up by 50MB,
>     > >> but when I try displaying the image using imshow(), my memory
>     usage goes up by around 500MB, each
>     > >> time.  If I close the figure and replot it, imshow() crashes.  I
>     don't know if I'm doing something
>     > >> wrong, or if it's a new or known bug.  I tried the same thing on
>     Linux and got the same result.
>     > >> Here's a transcript.
>     > >>
>     > >>       Welcome to pylab, a matplotlib-based Python environment.
>     > >>       For more information, type 'help(pylab)'.
>     > >>
>     > >> In [1]: import pyfits
>     > >>
>     > >> In [2]: from Tkinter import *
>     > >>
>     > >> In [3]: import tkFileDialog
>     > >>
>     > >> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>     > >>
>     > >> In [5]: imshow(image)
>     > >> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>     > >>
>     > >> In [6]: close()
>     > >>
>     > >> In [7]: imshow(image,origin='lower')
>     > >> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>     > >>
>     > >> In [8]: close()
>     > >>
>     > >> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>     > >> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>     > >>
>     > >> In [10]: Exception in Tkinter callback
>     > >> Traceback (most recent call last):
>     > >>       File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410,
>     in __call__
>     > >>         return self.func(*args)
>     > >>       File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495,
>     in callit
>     > >>         func(*args)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py",
>     line 263, in
>     > >> idle_draw
>     > >>         self.draw()
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py",
>     line 248, in draw
>     > >>         FigureCanvasAgg.draw(self)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py",
>     line 394, in draw
>     > >>         self.figure.draw(self.renderer)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55,
>     in draw_wrapper
>     > >>         draw(artist, renderer, *args, **kwargs)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798,
>     in draw
>     > >>         func(*args)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55,
>     in draw_wrapper
>     > >>         draw(artist, renderer, *args, **kwargs)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946,
>     in draw
>     > >>         a.draw(renderer)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55,
>     in draw_wrapper
>     > >>         draw(artist, renderer, *args, **kwargs)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354,
>     in draw
>     > >>         im = self.make_image(renderer.get_image_magnification())
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569,
>     in make_image
>     > >>         transformed_viewLim)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201,
>     in _get_unsampled_image
>     > >>         x = self.to_rgba(self._A, self._alpha)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in
>     to_rgba
>     > >>         x = self.norm(x)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820,
>     in __call__
>     > >>         result = (val-vmin) / (vmax-vmin)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in
>     __div__
>     > >>         return divide(self, other)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in
>     __call__
>     > >>         m |= filled(domain(da, db), True)
>     > >>       File
>     "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in
>     __call__
>     > >>         return umath.absolute(a) * self.tolerance>=
>     umath.absolute(b)
>     > >> MemoryError
>     > >>
>     > >>
>     > >> Thanks for any help,
>     > >> -robert
>     > >>
>     > >
>     > >
>     > > These are previous discussions on the issue:
>     > >
>     > >
>     <http://www.mail-archive.com/mat...@li.../msg14727.html>
>     > >
>     <http://www.mail-archive.com/mat...@li.../msg19815.html>
>     > >
>     <http://www.mail-archive.com/mat...@li.../msg19614.html>
>     > >
>     > > Christoph
>     > >
>     >
>     >  The first 2 discussions lead to suggestions of more memory on a
>     64-bit installation, but that
>     >  doesn't seem like a great solution.  I use other image processing
>     software (ImageJ and IDL) and
>     >  neither has any trouble with my images.  As I mentioned ImageJ
>     uses 1/10th the memory for the same
>     >  display, or about 1 byte of memory for 1 byte of image.  I think
>     matplotlib should aim for the same.
>     >     I also think it should free up memory when the image is closed,
>     but maybe I'm not doing the right
>     >  thing.  Is there something else I should be doing to free up memory?
>     >
>     >  Things are even worse with plot.
>     >
>     >  I'll file a bug report as Benjamin suggests.
>
>     Please file it in the "enhancement" category, not as a "bug".  The
>     difficulty that mpl has with images is the result of a design decision
>     long ago; as Christoph notes, mpl works primarily with float64 rgba,
>     which is horribly memory inefficient, but is very general and works fine
>     so long as the image is not too large.  uint8 is already used in some
>     circumstances.  I don't know how hard it would be to simply switch to
>     uint8 for images, or to make that an option, perhaps for all color
>     specification.  It may involve getting into arcana of the extension code
>     and the agg library, which is a daunting prospect for most of us,
>     certainly for me.  I agree entirely that a major reduction in memory
>     usage would be good; patches, or a mpl branch to achieve that, are
>     welcome.
>
>     Eric
>
>
> As a possible stop-gap solution, maybe we could consider the compressed
> arrays (carry)?
>
> I am not a fan of doing special cases, though, especially at the very
> base/core portions of the library.
>
> Ben Root
>

Forget about the patch; it is too crude and breaks things. My point was 
that it should be possible to have huge memory savings without touching 
agg internals. I'll take another look.

Christoph

Re: [Matplotlib-users] Trouble with imshow

[gary r. <gar...@gm...>]

Christoph, if you're looking at special casing uint8's, you might want
to keep in mind that uint16 greyscale images are also quite common as
camera outputs in experimental setups. I think that the solution to
this should ideally minimise memory usage for any greyscale image, be
it uint8, uint16, float32 or float64. i.e. avoiding conversion to RGBA
for any single-plane 2D array type would be best IMHO,

Gary R.

On Thu, Feb 3, 2011 at 5:38 PM, Robert Abiad <ab...@ss...> wrote:
>
>
> On 2/2/2011 6:06 PM, Eric Firing wrote:
>> On 02/02/2011 03:08 PM, Robert Abiad wrote:
>>> On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
>>>> On 2/2/2011 3:33 PM, Robert Abiad wrote:
>>>>> Hello All,
>>>>>
>>>>> I'm very new to python, so bear with me.
>>>>>
>>>>> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
>>>>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
>>>>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
>>>>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
>>>>> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
>>>>> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
>>>>> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
>>>>> Here's a transcript.
>>>>>
>>>>>        Welcome to pylab, a matplotlib-based Python environment.
>>>>>        For more information, type 'help(pylab)'.
>>>>>
>>>>> In [1]: import pyfits
>>>>>
>>>>> In [2]: from Tkinter import *
>>>>>
>>>>> In [3]: import tkFileDialog
>>>>>
>>>>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>>>>>
>>>>> In [5]: imshow(image)
>>>>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>>>>>
>>>>> In [6]: close()
>>>>>
>>>>> In [7]: imshow(image,origin='lower')
>>>>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>>>>>
>>>>> In [8]: close()
>>>>>
>>>>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>>>>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>>>>>
>>>>> In [10]: Exception in Tkinter callback
>>>>> Traceback (most recent call last):
>>>>>        File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>>>>>          return self.func(*args)
>>>>>        File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>>>>>          func(*args)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
>>>>> idle_draw
>>>>>          self.draw()
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>>>>>          FigureCanvasAgg.draw(self)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>>>>>          self.figure.draw(self.renderer)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>>          draw(artist, renderer, *args, **kwargs)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>>>>>          func(*args)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>>          draw(artist, renderer, *args, **kwargs)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>>>>>          a.draw(renderer)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>>          draw(artist, renderer, *args, **kwargs)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>>>>>          im = self.make_image(renderer.get_image_magnification())
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>>>>>          transformed_viewLim)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>>>>>          x = self.to_rgba(self._A, self._alpha)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>>>>>          x = self.norm(x)
>>>>>        File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>>>>>          result = (val-vmin) / (vmax-vmin)
>>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>>>>>          return divide(self, other)
>>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>>>>>          m |= filled(domain(da, db), True)
>>>>>        File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>>>>>          return umath.absolute(a) * self.tolerance>= umath.absolute(b)
>>>>> MemoryError
>>>>>
>>>>>
>>>>> Thanks for any help,
>>>>> -robert
>>>>>
>>>>
>>>> These are previous discussions on the issue:
>>>>
>>>> <http://www.mail-archive.com/mat...@li.../msg14727.html>
>>>> <http://www.mail-archive.com/mat...@li.../msg19815.html>
>>>> <http://www.mail-archive.com/mat...@li.../msg19614.html>
>>>>
>>>> Christoph
>>>>
>>> The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that
>>> doesn't seem like a great solution.  I use other image processing software (ImageJ and IDL) and
>>> neither has any trouble with my images.  As I mentioned ImageJ uses 1/10th the memory for the same
>>> display, or about 1 byte of memory for 1 byte of image.  I think matplotlib should aim for the same.
>>>     I also think it should free up memory when the image is closed, but maybe I'm not doing the right
>>> thing.  Is there something else I should be doing to free up memory?
>>>
>>> Things are even worse with plot.
>>>
>>> I'll file a bug report as Benjamin suggests.
>> Please file it in the "enhancement" category, not as a "bug".  The
>> difficulty that mpl has with images is the result of a design decision
>> long ago; as Christoph notes, mpl works primarily with float64 rgba,
>> which is horribly memory inefficient, but is very general and works fine
>> so long as the image is not too large.  uint8 is already used in some
>> circumstances.  I don't know how hard it would be to simply switch to
>> uint8 for images, or to make that an option, perhaps for all color
>> specification.  It may involve getting into arcana of the extension code
>> and the agg library, which is a daunting prospect for most of us,
>> certainly for me.  I agree entirely that a major reduction in memory
>> usage would be good; patches, or a mpl branch to achieve that, are welcome.
>>
>> Eric
> I'll put it in as an enhancement, but I'm still unsure if there is a bug in
> there as well.  Is there something I should be doing to clear memory after the
> first figure is closed other than close()?  I don't understand why memory usage
> grows each time I replot, but I'm pretty sure it isn't desireable behavior.  As
> I mentioned, this effect is worse with plot.
>
> So is this a bug or improper usage?
>
> As for how to implement a fix for memory usage, I'll let you folks figure that
> out.  But it seems that if I want a grayscale image, I could reduce memory usage
> by 4 if matplotlib could turn rgba into intensity.
>
> -robert
>
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>

Re: [Matplotlib-users] Trouble with imshow

[Christoph G. <cg...@uc...>]

As a workaround you could do your own normalization and color mapping 
and pass a uint8 RGB image to imshow. That avoids matplotlib's norm 
function. The following example saves almost 500 MB for plotting a 16 MB 
uint8 greyscale image, as compared to passing img directly to imshow:


import numpy
from matplotlib import pyplot

img = numpy.random.randint(0, 255, (4096, 4096)).astype('uint8')
lut = pyplot.cm.gray(numpy.arange(255), bytes=True)
rgb = lut.take(img, axis=0)
del img
pyplot.imshow(rgb)
pyplot.show()


Christoph


On 2/2/2011 11:00 PM, gary ruben wrote:
> Christoph, if you're looking at special casing uint8's, you might want
> to keep in mind that uint16 greyscale images are also quite common as
> camera outputs in experimental setups. I think that the solution to
> this should ideally minimise memory usage for any greyscale image, be
> it uint8, uint16, float32 or float64. i.e. avoiding conversion to RGBA
> for any single-plane 2D array type would be best IMHO,
>
> Gary R.
>
> On Thu, Feb 3, 2011 at 5:38 PM, Robert Abiad<ab...@ss...>  wrote:
>>
>>
>> On 2/2/2011 6:06 PM, Eric Firing wrote:
>>> On 02/02/2011 03:08 PM, Robert Abiad wrote:
>>>> On 2/2/2011 3:59 PM, Christoph Gohlke wrote:
>>>>> On 2/2/2011 3:33 PM, Robert Abiad wrote:
>>>>>> Hello All,
>>>>>>
>>>>>> I'm very new to python, so bear with me.
>>>>>>
>>>>>> I'd like to use python to do my image processing, but I'm running into behavior that doesn't make
>>>>>> sense to me.  I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest
>>>>>> versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy.  I'm loading in a fits file
>>>>>> that's 26 MB (~16 Mpixels).  When I load my image in ImageJ, I can see memory usage go up by 50MB,
>>>>>> but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each
>>>>>> time.  If I close the figure and replot it, imshow() crashes.  I don't know if I'm doing something
>>>>>> wrong, or if it's a new or known bug.  I tried the same thing on Linux and got the same result.
>>>>>> Here's a transcript.
>>>>>>
>>>>>>         Welcome to pylab, a matplotlib-based Python environment.
>>>>>>         For more information, type 'help(pylab)'.
>>>>>>
>>>>>> In [1]: import pyfits
>>>>>>
>>>>>> In [2]: from Tkinter import *
>>>>>>
>>>>>> In [3]: import tkFileDialog
>>>>>>
>>>>>> In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename())
>>>>>>
>>>>>> In [5]: imshow(image)
>>>>>> Out[5]:<matplotlib.image.AxesImage object at 0x03BCA170>
>>>>>>
>>>>>> In [6]: close()
>>>>>>
>>>>>> In [7]: imshow(image,origin='lower')
>>>>>> Out[7]:<matplotlib.image.AxesImage object at 0x0440E170>
>>>>>>
>>>>>> In [8]: close()
>>>>>>
>>>>>> In [9]: imshow(image[100:3600,100:3600],origin='lower')
>>>>>> Out[9]:<matplotlib.image.AxesImage object at 0x045D9FB0>
>>>>>>
>>>>>> In [10]: Exception in Tkinter callback
>>>>>> Traceback (most recent call last):
>>>>>>         File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 1410, in __call__
>>>>>>           return self.func(*args)
>>>>>>         File "C:\app\Python2.6\lib\lib-tk\Tkinter.py", line 495, in callit
>>>>>>           func(*args)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 263, in
>>>>>> idle_draw
>>>>>>           self.draw()
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 248, in draw
>>>>>>           FigureCanvasAgg.draw(self)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py", line 394, in draw
>>>>>>           self.figure.draw(self.renderer)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>>>           draw(artist, renderer, *args, **kwargs)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\figure.py", line 798, in draw
>>>>>>           func(*args)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>>>           draw(artist, renderer, *args, **kwargs)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\axes.py", line 1946, in draw
>>>>>>           a.draw(renderer)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
>>>>>>           draw(artist, renderer, *args, **kwargs)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 354, in draw
>>>>>>           im = self.make_image(renderer.get_image_magnification())
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 569, in make_image
>>>>>>           transformed_viewLim)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\image.py", line 201, in _get_unsampled_image
>>>>>>           x = self.to_rgba(self._A, self._alpha)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\cm.py", line 193, in to_rgba
>>>>>>           x = self.norm(x)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\matplotlib\colors.py", line 820, in __call__
>>>>>>           result = (val-vmin) / (vmax-vmin)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 3673, in __div__
>>>>>>           return divide(self, other)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 1077, in __call__
>>>>>>           m |= filled(domain(da, db), True)
>>>>>>         File "C:\app\Python2.6\lib\site-packages\numpy\ma\core.py", line 772, in __call__
>>>>>>           return umath.absolute(a) * self.tolerance>= umath.absolute(b)
>>>>>> MemoryError
>>>>>>
>>>>>>
>>>>>> Thanks for any help,
>>>>>> -robert
>>>>>>
>>>>>
>>>>> These are previous discussions on the issue:
>>>>>
>>>>> <http://www.mail-archive.com/mat...@li.../msg14727.html>
>>>>> <http://www.mail-archive.com/mat...@li.../msg19815.html>
>>>>> <http://www.mail-archive.com/mat...@li.../msg19614.html>
>>>>>
>>>>> Christoph
>>>>>
>>>> The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that
>>>> doesn't seem like a great solution.  I use other image processing software (ImageJ and IDL) and
>>>> neither has any trouble with my images.  As I mentioned ImageJ uses 1/10th the memory for the same
>>>> display, or about 1 byte of memory for 1 byte of image.  I think matplotlib should aim for the same.
>>>>      I also think it should free up memory when the image is closed, but maybe I'm not doing the right
>>>> thing.  Is there something else I should be doing to free up memory?
>>>>
>>>> Things are even worse with plot.
>>>>
>>>> I'll file a bug report as Benjamin suggests.
>>> Please file it in the "enhancement" category, not as a "bug".  The
>>> difficulty that mpl has with images is the result of a design decision
>>> long ago; as Christoph notes, mpl works primarily with float64 rgba,
>>> which is horribly memory inefficient, but is very general and works fine
>>> so long as the image is not too large.  uint8 is already used in some
>>> circumstances.  I don't know how hard it would be to simply switch to
>>> uint8 for images, or to make that an option, perhaps for all color
>>> specification.  It may involve getting into arcana of the extension code
>>> and the agg library, which is a daunting prospect for most of us,
>>> certainly for me.  I agree entirely that a major reduction in memory
>>> usage would be good; patches, or a mpl branch to achieve that, are welcome.
>>>
>>> Eric
>> I'll put it in as an enhancement, but I'm still unsure if there is a bug in
>> there as well.  Is there something I should be doing to clear memory after the
>> first figure is closed other than close()?  I don't understand why memory usage
>> grows each time I replot, but I'm pretty sure it isn't desireable behavior.  As
>> I mentioned, this effect is worse with plot.
>>
>> So is this a bug or improper usage?
>>
>> As for how to implement a fix for memory usage, I'll let you folks figure that
>> out.  But it seems that if I want a grayscale image, I could reduce memory usage
>> by 4 if matplotlib could turn rgba into intensity.
>>
>> -robert
>>
>> ------------------------------------------------------------------------------
>> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
>> Finally, a world-class log management solution at an even better price-free!
>> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
>> February 28th, so secure your free ArcSight Logger TODAY!
>> http://p.sf.net/sfu/arcsight-sfd2d
>> _______________________________________________
>> Matplotlib-users mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>>
>
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>

Re: [Matplotlib-users] Trouble with imshow

[Eric F. <ef...@ha...>]

On 02/02/2011 10:17 PM, Eric Firing wrote:
> On 02/02/2011 08:38 PM, Robert Abiad wrote:
>>
> [...]
>> I'll put it in as an enhancement, but I'm still unsure if there is a
>> bug in
>> there as well. Is there something I should be doing to clear memory
>> after the
>> first figure is closed other than close()? I don't understand why
>> memory usage
>> grows each time I replot, but I'm pretty sure it isn't desireable
>> behavior. As
>> I mentioned, this effect is worse with plot.
>>
>> So is this a bug or improper usage?
>
> I'm not quite sure, but I don't think there is a specifically matplotlib
> memory leak bug at work here. Are you using ipython, and if so, have you
> turned off the caching? In its default mode, ipython keeps lots of
> references, thereby keeping memory in use. Also, memory management and
> reporting can be a bit tricky and misleading.
>
> Nevertheless, the attached script may be illustrating the problem. Try
> running it from the command line as-is (maybe shorten the loop--it
> doesn't take 100 iterations to show the pattern) and then commenting out
> the line as indicated in the comment. It seems that if anything is done
> that adds ever so slightly to memory use while the figure is displayed,
> then when the figure is closed, its memory is not reused. I'm puzzled.

I wasn't thinking straight--there is no mystery and no memory leak. 
Ignore my example script referred to above.  It was saving rows of the z 
array, not single elements as I had intended, so of course memory use 
was growing substantially.

Eric

>
> Eric
>
>>
>> As for how to implement a fix for memory usage, I'll let you folks
>> figure that
>> out. But it seems that if I want a grayscale image, I could reduce
>> memory usage
>> by 4 if matplotlib could turn rgba into intensity.
>>
>> -robert
>
>
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
>
>
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users