Menu
▾
▴
Re: [Matplotlib-users] MemoryError with NetCDF
|
From: Eric F. <ef...@ha...> - 2012-09-07 17:04:53
|
On 2012/09/07 4:00 AM, Benjamin Root wrote:
>
>
> On Fri, Sep 7, 2012 at 9:49 AM, Shahar Shani-Kadmiel
> <ka...@po... <mailto:ka...@po...>> wrote:
>
> On Sep 7, 2012, at 4:25 PM, Benjamin Root wrote:
>
>>
>>
>> On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel
>> <ka...@po... <mailto:ka...@po...>> wrote:
>>
>> 1. an ipython session is invoked with qtconsole --pylab
>> 2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT
>> netCDF format (float) (COARDS-compliant) [DEFAULT]), approx.
>> 1.15 GB
>> 3. I then try to plot with imshow the data
>>
>> added below are the lines leading up to the error and the
>> error itself.
>>
>>
>> This is running on OS X 10.7.4 with a recently installed EPD 7.3.
>>
>>
>> {code}
>> from scipy.io <http://scipy.io/> import netcdf_file as netcdf
>> data =
>> netcdf('srtm_43_44_05_06_07_08.grd','r').variables['z'][::-1]
>>
>> fig, ax = subplots()
>>
>> data.shape
>> Out[5]: (24004, 12002)
>>
>> im = ax.imshow(data,
>> aspect=((data.shape[1])/float(data.shape[0])),
>> interpolation='none')
>> ---------------------------------------------------------------------------
>> MemoryError Traceback (most
>> recent call last)
>> <ipython-input-6-f92e4c4c63b5> in <module>()
>> ----> 1 im = ax.imshow(data,
>> aspect=((data.shape[1])/float(data.shape[0])),
>> interpolation='none')
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py
>> in imshow(self, X, cmap, norm, aspect, interpolation, alpha,
>> vmin, vmax, origin, extent, shape, filternorm, filterrad,
>> imlim, resample, url, **kwargs)
>> 6743 im.set_clim(vmin, vmax)
>> 6744 else:
>> -> 6745 im.autoscale_None()
>> 6746 im.set_url(url)
>> 6747
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py
>> in autoscale_None(self)
>> 281 if self._A is None:
>> 282 raise TypeError('You must first set_array
>> for mappable')
>> --> 283 self.norm.autoscale_None(self._A)
>> 284 self.changed()
>> 285
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py
>> in autoscale_None(self, A)
>> 889 ' autoscale only None-valued vmin or vmax'
>> 890 if self.vmin is None:
>> --> 891 self.vmin = ma.min(A)
>> 892 if self.vmax is None:
>> 893 self.vmax = ma.max(A)
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
>> in min(obj, axis, out, fill_value)
>> 5873 def min(obj, axis=None, out=None, fill_value=None):
>> 5874 try:
>> -> 5875 return obj.min(axis=axis,
>> fill_value=fill_value, out=out)
>> 5876 except (AttributeError, TypeError):
>> 5877 # If obj doesn't have a max method,
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
>> in min(self, axis, out, fill_value)
>> 5054 # No explicit output
>> 5055 if out is None:
>> -> 5056 result =
>> self.filled(fill_value).min(axis=axis, out=out).view(type(self))
>> 5057 if result.ndim:
>> 5058 # Set the mask
>>
>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc
>> in filled(self, fill_value)
>> 3388 return self._data
>> 3389 else:
>> -> 3390 result = self._data.copy()
>> 3391 try:
>> 3392 np.putmask(result, m, fill_value)
>>
>> MemoryError:
>> {/code}
>>
>>
>> This is more a NumPy issue than anything else. We need to know
>> the min and the max of the array in order to automatically scale
>> the colormap for display. Therefore, we query the array object
>> for its min/max. Because we support masked arrays, the array is
>> first cast as a masked array, and then these queries are done.
>>
>> It appears that in numpy's masked array module, it calculates the
>> array's min by making a copy of itself first. I would have
>> figured that it would have done its task differently. In the
>> meantime, I suspect you can work around this problem by explicitly
>> setting the vmin/vmax keyword arguments to imshow if you know
>> them. Therefore, there should be no need to determine the array's
>> min/max in this inefficient manner.
>>
>> Ben Root
>>
>
> Hi Ben,
> I tried adding vmin & vmax the the imshow call but I still get a
> MemoryError.
> The grid file is 1.15 GB and I have ~4.5 out of 8 GB of memory
> available when I launch ipython, 3.5 when I execute imshow and 2
> when I execute plt.draw().
mpl simply is not designed for image-type operations on huge arrays,
vastly larger than what can be displayed. It is up to the user to
down-sample or otherwise reduce the size of the array fed to imshow.
>
>
> Well, it looks like setting vmin/vmax helped, because your traceback
Did a new traceback get posted in a message that was not sent to the
list? I found only the original traceback.
> shows that the code made significant progress. The issue here is that
> the process_value() method doesn't make a lot of sense. I am not sure
> what is the rationale behind its logic. Hopefully, someone else can
> chime in with an explanation of what is going on.
Normalization has to handle all sorts of inputs--masked or not, all
sorts of numbers, scalar or array--and it is much easier to do this
efficiently if all these possibilities are reduced to a very few at the
start. Specifically, it needs to supply a copy of the input (so that
normalization doesn't change the original) in a floating point masked
array, using float32 if possible for space efficiency. It needs to keep
track of whether the input was a scalar, so that normalization can
return a scalar when given a scalar input.
Eric
>
> In the meantime, are you using a 32 or 64-bit machine?
>
> Ben Root
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
|
