Re: [Matplotlib-users] MemoryError with NetCDF

[Benjamin R. <ben...@ou...>]

On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel <ka...@po...
> wrote:

> 1. an ipython session is invoked with qtconsole --pylab
> 2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT netCDF
> format (float)  (COARDS-compliant) [DEFAULT]), approx. 1.15 GB
> 3. I then try to plot with imshow the data
>
> added below are the lines leading up to the error and the error itself.
>
>
> This is running on OS X 10.7.4 with a recently installed EPD 7.3.
>
>
> {code}
> from scipy.io import netcdf_file as netcdf
> data = netcdf('srtm_43_44_05_06_07_08.grd','r').variables['z'][::-1]
>
> fig, ax = subplots()
>
> data.shape
> Out[5]: (24004, 12002)
>
> im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])),
> interpolation='none')
> ---------------------------------------------------------------------------
> MemoryError                               Traceback (most recent call last)
> <ipython-input-6-f92e4c4c63b5> in <module>()
> ----> 1 im = ax.imshow(data,
> aspect=((data.shape[1])/float(data.shape[0])), interpolation='none')
>
> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py
> in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax,
> origin, extent, shape, filternorm, filterrad, imlim, resample, url,
> **kwargs)
>    6743             im.set_clim(vmin, vmax)
>    6744         else:
> -> 6745             im.autoscale_None()
>    6746         im.set_url(url)
>    6747
>
> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py
> in autoscale_None(self)
>     281         if self._A is None:
>     282             raise TypeError('You must first set_array for
> mappable')
> --> 283         self.norm.autoscale_None(self._A)
>     284         self.changed()
>     285
>
> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py
> in autoscale_None(self, A)
>     889         ' autoscale only None-valued vmin or vmax'
>     890         if self.vmin is None:
> --> 891             self.vmin = ma.min(A)
>     892         if self.vmax is None:
>     893             self.vmax = ma.max(A)
>
> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
> in min(obj, axis, out, fill_value)
>    5873 def min(obj, axis=None, out=None, fill_value=None):
>    5874     try:
> -> 5875         return obj.min(axis=axis, fill_value=fill_value, out=out)
>    5876     except (AttributeError, TypeError):
>    5877         # If obj doesn't have a max method,
>
> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
> in min(self, axis, out, fill_value)
>    5054         # No explicit output
>    5055         if out is None:
> -> 5056             result = self.filled(fill_value).min(axis=axis,
> out=out).view(type(self))
>    5057             if result.ndim:
>    5058                 # Set the mask
>
> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
> in filled(self, fill_value)
>    3388             return self._data
>    3389         else:
> -> 3390             result = self._data.copy()
>    3391             try:
>    3392                 np.putmask(result, m, fill_value)
>
> MemoryError:
> {/code}
>

This is more a NumPy issue than anything else.  We need to know the min and
the max of the array in order to automatically scale the colormap for
display.  Therefore, we query the array object for its min/max.  Because we
support masked arrays, the array is first cast as a masked array, and then
these queries are done.

It appears that in numpy's masked array module, it calculates the array's
min by making a copy of itself first.  I would have figured that it would
have done its task differently.  In the meantime, I suspect you can work
around this problem by explicitly setting the vmin/vmax keyword arguments
to imshow if you know them.  Therefore, there should be no need to
determine the array's min/max in this inefficient manner.

Ben Root