Re: [Matplotlib-users] memory error - netcdf plots

[Jeff W. <js...@fa...>]

On 10/18/11 8:55 PM, questions anon wrote:
> Thanks Jeff, that certainly speeds it up! But when I take them out of 
> the loop and place them elsewhere they are no longer added to the map.
> Is there someway I can call them in the loop but still get it to run 
> quickly?
> Thanks

Just the Basemap instance creation and the transformation of coordinates 
to projection space should be hoisted out of the loop

                 map = 
Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
                               
llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
                 x,y=map(*N.meshgrid(LON,LAT))

you can leave the other statements in.

If you still have memory issues, bring the readshapefile call out, and 
draw the shapes whose coordinates are stored in the instance variable 
map.DSE_REGIONS manually in the loop.

-Jeff
>
> On Fri, Oct 14, 2011 at 10:54 PM, Jeff Whitaker <js...@fa... 
> <mailto:js...@fa...>> wrote:
>
>     On 10/12/11 8:20 PM, questions anon wrote:
>>     Hi All,
>>     I keep receiving a memory error when processing many netcdf
>>     files. I assumed it had something to do with how I loop things
>>     and maybe needed to close things off properly but I recently
>>     received an error that made me think it might be because of
>>     matplotlib.
>>
>>     In the code below I am looping through a bunch of netcdf files
>>     (each file is hourly data for one month) and within each netcdf
>>     file I am outputting a *png file every three hours. This works
>>     for one netcdf file (therefore one month) but when it begins to
>>     process the next netcdf file I receive a memory error (see
>>     below). Since I have tidied some of my code up it seems to
>>     process partly into the second file but then I still receive the
>>     memory error.
>>     I have tried a few suggestions such as:
>>     -Combining the dataset using MFDataset (using NETCDF4) is not an
>>     option because the files do not have unlimited dimension.
>>     - gc.collect() but that just results in a /GEOS_ERROR: bad
>>     allocation error/.
>>     -only open LAT and LON once (which worked)
>>
>>     System Details:
>>     Python 2.7.2 |EPD 7.1-2 (32-bit)| (default, Jul  3 2011,
>>     15:13:59) [MSC v.1500 32 bit (Intel)] on win32
>>
>>     Any feedback will be greatly appreciated as I seem to keep ending
>>     up with memory errors when working with netcdf files this even
>>     happens if I am using a much better computer.
>>
>>     *Most recent error: *
>>     Traceback (most recent call last):
>>       File
>>     "C:\plot_netcdf_merc_multiplot_across_multifolders_TSFC.py", line
>>     78, in <module>
>>         plt.savefig((os.path.join(outputfolder,
>>     'TSFC'+date_string+'UTC.png')))
>>       File "C:\Python27\lib\site-packages\matplotlib\pyplot.py", line
>>     363, in savefig
>>         return fig.savefig(*args, **kwargs)
>>       File "C:\Python27\lib\site-packages\matplotlib\figure.py", line
>>     1084, in savefig
>>         self.canvas.print_figure(*args, **kwargs)
>>       File
>>     "C:\Python27\lib\site-packages\matplotlib\backends\backend_wxagg.py",
>>     line 100, in print_figure
>>         FigureCanvasAgg.print_figure(self, filename, *args, **kwargs)
>>       File
>>     "C:\Python27\lib\site-packages\matplotlib\backend_bases.py", line
>>     1923, in print_figure
>>         **kwargs)
>>       File
>>     "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>>     line 438, in print_png
>>         FigureCanvasAgg.draw(self)
>>       File
>>     "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>>     line 393, in draw
>>         self.renderer = self.get_renderer()
>>       File
>>     "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>>     line 404, in get_renderer
>>         self.renderer = RendererAgg(w, h, self.figure.dpi)
>>       File
>>     "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>>     line 59, in __init__
>>         self._renderer = _RendererAgg(int(width), int(height), dpi,
>>     debug=False)
>>     RuntimeError: Could not allocate memory for image
>>
>>     *Error when I added gc.collect()*
>>     GEOS_ERROR: bad allocation
>>
>>     *Old error (before adding gc.collect() )*
>>     /Traceback (most recent call last):
>>       File
>>     "d:/plot_netcdf_merc_multiplot_across_multifolders__memoryerror.py",
>>     line 44, in <module>
>>         TSFC=ncfile.variables['T_SFC'][1::3]
>>       File "netCDF4.pyx", line 2473, in netCDF4.Variable.__getitem__
>>     (netCDF4.c:23094)
>>     MemoryError/
>>
>>
>>
>>     from netCDF4 import Dataset
>>     import numpy as N
>>     import matplotlib.pyplot as plt
>>     from mpl_toolkits.basemap import Basemap
>>     from netcdftime import utime
>>     from datetime import datetime
>>     import os
>>     import gc
>>
>>
>>     shapefile1="E:/
>>
>>         griddeddatasamples/GIS/DSE_REGIONS"
>>         MainFolder=r"E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/"
>>         OutputFolder=r"E:/griddeddatasamples/GriddedData/OutputsforValidation"
>>         fileforlatlon=Dataset("E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/TSFC_1974_01/IDZ00026_VIC_ADFD_T_SFC.nc",
>>         'r+', 'NETCDF4')
>>         LAT=fileforlatlon.variables['latitude'][:]
>>         LON=fileforlatlon.variables['longitude'][:]
>>
>>         for (path, dirs, files) in os.walk(MainFolder):
>>             for dir in dirs:
>>                 print dir
>>             path=path+'/'
>>             for ncfile in files:
>>                 if ncfile[-3:]=='.nc':
>>                     print "dealing with ncfiles:", ncfile
>>                     ncfile=os.path.join(path,ncfile)
>>                     ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
>>                     TSFC=ncfile.variables['T_SFC'][1::3]
>>                     TIME=ncfile.variables['time'][1::3]
>>                     ncfile.close()
>>                     gc.collect()
>>
>>                     for TSFC, TIME in zip((TSFC[:]),(TIME[:])):
>>                         cdftime=utime('seconds since 1970-01-01
>>         00:00:00')
>>                         ncfiletime=cdftime.num2date(TIME)
>>                         print ncfiletime
>>                         timestr=str(ncfiletime)
>>                         d = datetime.strptime(timestr, '%Y-%m-%d
>>         %H:%M:%S')
>>                         date_string = d.strftime('%Y%m%d_%H%M')
>>
>>                         map =
>>         Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>>                                      
>>         llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>>                         x,y=map(*N.meshgrid(LON,LAT))
>>                         map.drawcoastlines(linewidth=0.5)
>>                         map.readshapefile(shapefile1, 'DSE_REGIONS')
>>                         map.drawstates()
>>
>>                         plt.title('Surface temperature at %s
>>         UTC'%ncfiletime)
>>                         ticks=[-5,0,5,10,15,20,25,30,35,40,45,50]
>>                         CS = map.contourf(x,y,TSFC, ticks,
>>         cmap=plt.cm.jet)
>>                         l,b,w,h =0.1,0.1,0.8,0.8
>>                         cax = plt.axes([l+w+0.025, b, 0.025, h], )
>>                         cbar=plt.colorbar(CS, cax=cax, drawedges=True)
>>
>>                         plt.savefig((os.path.join(OutputFolder,
>>         'TSFC'+date_string+'UTC.png')))
>>                         plt.close()
>>                         gc.collect()
>>
>
>     Try moving these lines
>
>
>                     map =
>     Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>                                  
>     llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>                     x,y=map(*N.meshgrid(LON,LAT))
>                     map.drawcoastlines(linewidth=0.5)
>                     map.readshapefile(shapefile1, 'DSE_REGIONS')
>                     map.drawstates()
>
>     out of the loop.
>
>     -Jeff
>
>