Memory Requirements for Adding a Record Dimension

[silverhawk]

Hi,

I have a netCDF file that I'm trying to change a fixed dimension to a record dimension and my process is being killed for too much memory usage.

I saw the Memory Requirements listed here: http://nco.sourceforge.net/nco.html#Memory-Requirements but I don't see my situation specified.

I am using the following ncks command to perform the operation:

ncks -O --mk_rec_dmn y fileIn.nc fileOut.nc

What is the max memory that I should expect to need to perform that operation?
I thought it would likely be 2 times Memory required to store the largest variable to be processed in an input file but I have more than double that available and it's still running out.

Thanks!

[Charlie Zender]

Yes, max memory for this should be ~2x sizeof(largest variable). We have seen issues where Windows bails with much less memory.

[silverhawk]

Do you mind trying to run that command against the attached file for me? I'm trying on RHEL with what should be plenty of memory and ncks just runs until it consumes all the memory available and the OS kills it.

[Charlie Zender]

This 19 MB file requires ~500 MB RAM for this command. Apparently the netCDF4 library requires extra overhead to decompress the data. The chunking might also be an issue. This is actually a huge dataset that is very efficiently compressed, and re-writing requires decompression/recompression.

 dimensions:
    comment_meta_data_len = 1420 ;
    t = 1 ;
    x = 25880 ;
    xml_meta_data_len = 67942 ;
    y = 18080 ;
    z = 1 ;

  variables:
    byte BaseReflMosaic(t,z,y,x) ;
      BaseReflMosaic:_FillValue = -128b ; // byte
      BaseReflMosaic:standard_name = "equivalent_reflectivity_factor" ; // char
      BaseReflMosaic:long_name = "Base reflectivity mosaic" ; // char
      BaseReflMosaic:class_name = "DZ" ; // char
      BaseReflMosaic:product_identifier = "http://www.faa.gov/ontology/wx/1.0/nwpWeatherProducts.owl#BaseReflMosaic" ; // char
      BaseReflMosaic:product_name = "BaseReflMosaic" ; // char
      BaseReflMosaic:ancillary_variables = "BaseReflMosaicFlags SurfacePrecipPhase" ; // char
      BaseReflMosaic:units = "dBZ" ; // char
      BaseReflMosaic:grid_mapping = "grid_mapping" ; // char
      BaseReflMosaic:scale_factor = 0.5 ; // double
      BaseReflMosaic:add_offset = 31.5 ; // double
      BaseReflMosaic:valid_range = -127b, 126b ; // byte
      BaseReflMosaic:color_mappings = "color_mapping_traffic_manager_standard_rain color_mapping_traffic_manager_standard_mixed color_mapping_traffic_manager_standard_snow color_mapping_traffic_manager_weak_rain color_mapping_traffic_manager_weak_mixed color_mapping_traffic_manager_weak_snow color_mapping_terminal_controller_live color_mapping_enroute_controller_demo color_mapping_enroute_controller color_mapping_meteorologist_rain color_mapping_meteorologist_mixed color_mapping_meteorologist_snow" ; // char
      BaseReflMosaic:_Storage = "chunked" ; // char
      BaseReflMosaic:_ChunkSizes = 1, 1, 226, 647 ; // int
      BaseReflMosaic:_DeflateLevel = 6 ; // int

[Charlie Zender]

Actually the max required RAM is only ~1xsizeof(uncompressed largest variable). Yours is 25880x18080=467910400 B

[silverhawk]

Charlie - thanks for the explanation. Any ideas why the ncks process is literally taking up over 8GB of memory before being killed for me? Were you able to successfully execute it?

[Charlie Zender]

It executes fine on my Mac and Linux machines with ~500 MB RAM required. Unclear why 8 GB for you. Try instead
ncks -O -t 0 --mk_rec_dmn y fileIn.nc fileOut.nc

[silverhawk]

-t doesn't seem to be an option for me

[root@rhelvm sf_VM_Shared]# ncks -O -t --mk_rec_dmn y fileIn.nc fileOut.nc
ncks: invalid option -- 't'
ncks Command line options cheatsheet (full details at http://nco.sf.net/nco.html#ncks):
ncks [-3] [-4] [-5] [-6] [-7] [-A] [-a] [-b fl_bnr] [--bfr sz] [-C] [-c] [--cdl] [--cnk_byt sz] [--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz] [-D nco_dbg_lvl] [-d ...] [-F] [--fix_rec_dmn dim] [--fl_fmt fmt] [-G grp:lvl] [-g ...] [-H] [-h] [--hdn] [--hdr_pad nbr] [-L lvl] [-l path] [-M] [-m] [--md5_dgs] [--md5_wrt] [--mk_rec_dmn dim] [--msa] [--no_blank] [--no_tmp_fl] [-O] [-o out.nc] [-P] [-p path] [-Q] [-q] [-R] [-r] [--rad] [--ram_all] [-s format] [-u] [--unn] [-v ...] [-X box] [-x] [--xml] [--xml_no_loc] [--xml_spr_chr sng] [--xml_spr_nmr sng] in.nc [[out.nc]]

[Charlie Zender]

You must have an old (and possibly buggy) version of NCO. Suggest upgrading. Command works fine on recent NCO's. Try a conda install if you need but do not have root permissions.

[silverhawk]

Wow - you're right. Eventhough I installed it less than a month ago I'm only on version 4.4.4. I'll have to try the conda install instead of wherever I got these rpms from.

The file I had uploaded was the original file, but it looks like the one that was actually failing is a hyperslab of that which had the record dimension set to 'x' already. And ncks was trying to change it to y. Hopefully there was a bug that was fixed between. 4.4.4 and 4.6.6 which handles that.

Thanks for you're fast responses as always!

[silverhawk]

My workaround prior to re-installing the nco tools was to add an extra command in that set x to a fixed dimension and then the settting of y to the record dimension works fine

[silverhawk]

Charlie,

Just a quick follow-up question for you. Why do you just multiple the dimensions when you calculate the uncompressed size? If I use ncdump to extract the BaseReflMosaic variable I actually get a file that's almost 1.9GB.

[Charlie Zender]

I computed size in RAM which is always much less than size of formatted ASCII text file on disk.