[nco-announce] netCDF Operators NCO version 4.4.7 are ready
Command-line operators for netCDF and HDF files
Brought to you by:
zender
Menu
▾
▴
[nco-announce] netCDF Operators NCO version 4.4.7 are ready
|
From: Charlie Z. <ze...@uc...> - 2014-11-26 21:20:34
|
The netCDF Operators NCO version 4.4.7 are ready. http://nco.sf.net (Homepage) http://dust.ess.uci.edu/nco (Homepage "mirror") This stabilization release also contains useful new features that assist users with chunking policies, hyperslabbing, and creating record variables. First, hyperslab specifications now use Python-consistent indexing (negative indices) to indicate the last elements of a dimension. Second, chunking behavior has been revamped. Overall the chunking algorithms are much more robust and optimized. See below for the user-visible changes. Of note is the new balanced chunking map named for Russ Rew. Finally, ncap2 can create record variables. Work on NCO 4.4.8 is underway, still focused on stability and speed. This includes more netCDF4 mop-up and new chunking features. Enjoy, Charlie NEW FEATURES (full details always in ChangeLog): A. Hyperslab specifications now accept -1 (negative one) to indicate the last element of a dimension. This is consistent with Python syntax, unlike NCO's previous behavior (from 20120709--20141001) where negative integers indicated the number of elements from the last (-1 was penultimate, -0 was ultimate). Now -1 is last element, -2 is penultimate element, and -N is the first element. NCO warns users who may expect the old behavior. ncks -d lon,-1 in.nc out.nc # last longitude ncks -d lon,-3,-1 in.nc out.nc # last three longitudes http://nco.sf.net/nco.html#hyp B. ncap2 creates record dimension when dimension size is negative: ncap2 -s 'defdim("time",10)' in.nc out.nc # Fixed-size dimension ncap2 -s 'defdim("time",-10)' in.nc out.nc # Record dimension Formerly ncap2 had no way to create new record dimensions. http://nco.sf.net/nco.html#ncap2 C. A new chunking map, --cnk_map=rew, implements a chunking strategy that balances common 1-D (e.g., timeseries) and 2-D (e.g., geographic slices) access patterns to 3-D (time, lat, lon) data. Name in honor of its (and netCDF's) developer Russ Rew: http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes For now, this map only applies to 3-D variables. We hope to include a version generalized to N-D variables in the next release. ncks --cnk_map=rew in.nc out.nc # Use Rew's balanced chunking http://nco.sf.net/nco.html#cnk D. A new chunking map, --cnk_map=nc4, implements the netCDF4 default chunking map as implemented in the netCDF library used to build NCO. Formerly, NCO had no way to explicitly resort to netCDF4 defaults. (netCDF4 and NCO chunking defaults differ). This option allows users to change the chunking of a dataset to what it would be if it were created from scratch by ncgen without any specific chunking options. ncks --cnk_map=nc4 in.nc out.nc # Use netCDF4 default chunking http://nco.sf.net/nco.html#cnk E. NCO's default chunking policy now preserves existing chunking. Previously NCO would use its favorite chunking policy (which variables to chunk) and map (how to chunk those variables) on all input files unless the user explicitly specified other chunking options. Thus a chain of processing commands had to explicitly specify chunking parameters in each command otherwise the chunksizes could be reset to a shape not optimal for the anticipated access patterns. Now once a file is optimally chunked, no further NCO operations on that file need specify chunking options (since they are preserved by default). ncks in.nc out.nc # Defaults to --cnk_plc=xst, --cnk_map=xst ncks --cnk_map=xst --cnk_plc=xst in.nc out.nc # Same effect http://nco.sf.net/nco.html#cnk F. The minimum size of variables to chunk may now be specified with --cnk_min=var_sz, where var_sz is the minimum size in bytes (not elements) of variables to chunk. This threshold is intended to restrict use of chunking to variables for which it is efficient. By default this minimum variable size for chunking is 8192 B. Formerly, NCO would chunk all arrays of any size. To obtain that behavior now users must specifiy --cnk_min=1, so that arrays of any size will be chunked. To obtain a system-dependent minimum size, set cnk_min=0 and then NCO will compute the minimum size as twice the system blocksize (when available) and 8192 B otherwise. ncks in.nc out.nc # Minimum variable size to chunk is 8192 B ncks --cnk_min=1 in.nc out.nc # Minimum size is 1 B ncks --cnk_min=0 in.nc out.nc # Minimum size is twice blocksize http://nco.sf.net/nco.html#cnk BUG FIXES: A. ncrename has known problems when renaming netCDF4 coordinates. These problems will go away when Unidata issues a fix to the underlying netCDF library. Thanks to Parker Allen for reporting. http://nco.sf.net#bug_nc4_rename B. Auxiliary coordinate hyperslabbing (with -X) was inadvertently turned-off for 1-D variables. Now it works fine with 1-D vars. Fixed by Pedro Vicente. Moreover, files without a "Conventions=CF-1.X" attribute are now treated as if they had the attribute to exploit -X functionality. C. Auxiliary coordinate hyperslabbing (with -X) was broken for variables with non-coordinate dimensions. Fixed by Pedro Vicente. Thanks to romzie04 for alerting us to these -X issues. D. Fixed ncks bug when built on netCDF < 4.3.1 due to error in handling NULL arguments to nco_inq_format_extended() stub. Also fix printing problem with strncpy(). Only known to affect with strict initialization like clang. KNOWN PROBLEMS DUE TO NCO: This section of ANNOUNCE reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.4.7 built/tested with netCDF 4.3.3-rc2 (20141112) on top of HDF5 hdf5-1.8.13 with: cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or- cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism A. NOT YET FIXED (NCO problem) Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments Demonstration: ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc ncks -m -C -v att_var ~/foo.nc 20130724: Verified problem still exists TODO nco1102 Cause: NCO parsing of ncatted arguments is not sophisticated enough to handle arrays of NC_STRINGS with embedded delimiters. B. NOT YET FIXED (NCO problem?) ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions Demonstration: ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc 20140826: Verified problem still exists 20140619: Problem reported by rmla Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD? Workaround: Convert to fixed dimensions then hyperslab KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS: A. NOT YET FIXED (netCDF4 problem) Renaming netCDF4 coordinate variables or dimensions "succeeds" but corrupts (sets to _FillValue) values in the output dataset. Full description here http://nco.sf.net#bug_nc4_rename Demonstration with netCDF <= 4.3.2: ncrename -O -v time,newrec ~/nco/data/in_grp.nc ~/foo.nc ncks --cdl -g // -v newrec -d time,0 -C ~/foo.nc 20141007: Problem reported by Parker Norton 20141008: Problem reported to Unidata 20141010: Verified by Unidata. 20141112: Verified problem still exists Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-177 Workaround: Convert to netCDF3, rename, convert back to netCDF4 B. NOT YET FIXED (netCDF4 or HDF5 problem?) Specifying strided hyperslab on large netCDF4 datasets leads to slowdown or failure with recent netCDF versions. Demonstration with NCO <= 4.4.5: time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc Demonstration with NCL: time ncl < ~/nco/data/ncl.ncl 20140718: Problem reported by Parker Norton 20140826: Verified problem still exists 20140930: Finish NCO workaround for problem Cause: Slow algorithm in nc_var_gets()? Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets()) Workaround #2: Convert file to netCDF3 first, then use stride C. NOT YET FIXED (would require DAP protocol change?) Unable to retrieve contents of variables including period '.' in name Periods are legal characters in netCDF variable names. Metadata are returned successfully, data are not. DAP non-transparency: Works locally, fails through DAP server. Demonstration: ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable 20130724: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts. NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string. Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47 D. NOT YET FIXED (would require DAP protocol change) Correctly read scalar characters over DAP. DAP non-transparency: Works locally, fails through DAP server. Problem, IMHO, is with DAP definition/protocol Demonstration: ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc 20120801: Verified problem still exists Bug report not filed Cause: DAP translates scalar characters into 64-element (this dimension is user-configurable, but still...), NUL-terminated strings so MD5 agreement fails "Sticky" reminders: A. Pre-built Debian Sid & Ubuntu packages: http://nco.sf.net#debian B. Pre-built Fedora and CentOS RPMs: http://nco.sf.net#rpm C. Pre-built Mac binaries: http://nco.sf.net#mac D. Pre-built Windows (native) and Cygwin binaries: http://nco.sf.net#windows E. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., HDF4: AMSR MERRA MODIS ... HDF5: GLAS ICESat Mabel SBUV ... HDF-EOS5: AURA HIRDLS OMI ... -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'( |