Re: [Matplotlib-users] MemoryError with NetCDF

[Benjamin R. <ben...@ou...>]

On Fri, Sep 7, 2012 at 9:49 AM, Shahar Shani-Kadmiel <ka...@po...
> wrote:

> On Sep 7, 2012, at 4:25 PM, Benjamin Root wrote:
>
>
>
> On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel <
> ka...@po...> wrote:
>
>> 1. an ipython session is invoked with qtconsole --pylab
>> 2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT netCDF
>> format (float)  (COARDS-compliant) [DEFAULT]), approx. 1.15 GB
>> 3. I then try to plot with imshow the data
>>
>> added below are the lines leading up to the error and the error itself.
>>
>>
>> This is running on OS X 10.7.4 with a recently installed EPD 7.3.
>>
>>
>> {code}
>> from scipy.io import netcdf_file as netcdf
>> data = netcdf('srtm_43_44_05_06_07_08.grd','r').variables['z'][::-1]
>>
>> fig, ax = subplots()
>>
>> data.shape
>> Out[5]: (24004, 12002)
>>
>> im = ax.imshow(data, aspect=((data.shape[1])/float(data.shape[0])),
>> interpolation='none')
>>
>> ---------------------------------------------------------------------------
>> MemoryError                               Traceback (most recent call
>> last)
>> <ipython-input-6-f92e4c4c63b5> in <module>()
>> ----> 1 im = ax.imshow(data,
>> aspect=((data.shape[1])/float(data.shape[0])), interpolation='none')
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py
>> in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax,
>> origin, extent, shape, filternorm, filterrad, imlim, resample, url,
>> **kwargs)
>>    6743             im.set_clim(vmin, vmax)
>>    6744         else:
>> -> 6745             im.autoscale_None()
>>    6746         im.set_url(url)
>>    6747
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py
>> in autoscale_None(self)
>>     281         if self._A is None:
>>     282             raise TypeError('You must first set_array for
>> mappable')
>> --> 283         self.norm.autoscale_None(self._A)
>>     284         self.changed()
>>     285
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py
>> in autoscale_None(self, A)
>>     889         ' autoscale only None-valued vmin or vmax'
>>     890         if self.vmin is None:
>> --> 891             self.vmin = ma.min(A)
>>     892         if self.vmax is None:
>>     893             self.vmax = ma.max(A)
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
>> in min(obj, axis, out, fill_value)
>>    5873 def min(obj, axis=None, out=None, fill_value=None):
>>    5874     try:
>> -> 5875         return obj.min(axis=axis, fill_value=fill_value, out=out)
>>    5876     except (AttributeError, TypeError):
>>    5877         # If obj doesn't have a max method,
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
>> in min(self, axis, out, fill_value)
>>    5054         # No explicit output
>>    5055         if out is None:
>> -> 5056             result = self.filled(fill_value).min(axis=axis,
>> out=out).view(type(self))
>>    5057             if result.ndim:
>>    5058                 # Set the mask
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
>> in filled(self, fill_value)
>>    3388             return self._data
>>    3389         else:
>> -> 3390             result = self._data.copy()
>>    3391             try:
>>    3392                 np.putmask(result, m, fill_value)
>>
>> MemoryError:
>> {/code}
>>
>
> This is more a NumPy issue than anything else.  We need to know the min
> and the max of the array in order to automatically scale the colormap for
> display.  Therefore, we query the array object for its min/max.  Because we
> support masked arrays, the array is first cast as a masked array, and then
> these queries are done.
>
> It appears that in numpy's masked array module, it calculates the array's
> min by making a copy of itself first.  I would have figured that it would
> have done its task differently.  In the meantime, I suspect you can work
> around this problem by explicitly setting the vmin/vmax keyword arguments
> to imshow if you know them.  Therefore, there should be no need to
> determine the array's min/max in this inefficient manner.
>
> Ben Root
>
>
> Hi Ben,
> I tried adding vmin & vmax the the imshow call but I still get a
> MemoryError.
> The grid file is 1.15 GB and I have ~4.5 out of 8 GB of memory available
> when I launch ipython, 3.5 when I execute imshow and 2 when I execute
> plt.draw().
>

Well, it looks like setting vmin/vmax helped, because your traceback shows
that the code made significant progress.  The issue here is that the
process_value() method doesn't make a lot of sense.  I am not sure what is
the rationale behind its logic.  Hopefully, someone else can chime in with
an explanation of what is going on.

In the meantime, are you using a 32 or 64-bit machine?

Ben Root