nco-announce

[Charlie Z. <ze...@uc...>]

The netCDF Operators NCO version 4.3.8 are ready.

http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")

ncatted now supports groups. That's the main new feature.
This completes NCO's migration from flat- to group-oriented files.
All core operators are now "group-aware" and support netCDF4 groups.
Most relevant options may support full or partial group paths.
Remaining options that do not will be migrated as time permits.

The exception to this is ncap2. Porting ncap2 to support groups
will be undertaken as resources permit. The main ncap2 developer
is a volunteer for whom we are trying to secure funding.

Work on NCO 4.3.9 is underway and includes improved netCDF4 support
for various options (all -X, ncwa -w and -m), other netCDF4 mop-up,
and improved support for HDF4 files. And the operator nces, which
we will debut/describe at AGU (Friday 12/13 11:35 AM Moscone West 2010).

Enjoy,
Charlie

"New stuff" in 4.3.8 summary (full details always in ChangeLog):

NEW FEATURES:

A. ncatted accepts full group paths to variables and groups.
   Previously ncatted worked only on the root group of netCDF4 files.
   Now you can edit all attributes in group files, in all
   permutations, i.e., for variables, for groups and the root group.
   ncatted -a units,/g1/lon,o,c,"degrees_west" in_grp.nc
   ncatted -a new_grp_att,group,c,sng,"new group attribute" in_grp.nc
   http://nco.sf.net/nco.html#xmp_ncatted

B. Add -7 switch to write netCDF4-classic files.
   This is simply a synonym for --fl_fmt=netcdf4_classic.
   The reason (and mnemonic) for -7 is that NETCDF4_CLASSIC files
   include great features of both netCDF3 (compatibility) and
   netCDF4 (compression, chunking) and, well, 3+4=7 :)
   Anyway, -7 is much easier to remember/type than the alternative.
   http://nco.sf.net/nco.html#fl_fmt

BUG FIXES:

A. Reduced namespace conflicts with other shared libraries

B. Added missing --op_typ/--operation switches to ncra

KNOWN ISSUES NOT YET FIXED:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.3.7 built/tested with netCDF
   4.3.1-rc4 snapshot 20131106 on top of HDF5 hdf5-1.8.9 with these
methods:

   cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
   cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism

A. NOT YET FIXED
   netCDF4 library fails when renaming dimension and variable using
   that dimension, in either order. Works fine with netCDF3.
   Hence coordinate renaming does not work with netCDF4 files.
   Problem with netCDF4 library implementation.

   Demonstration:
   ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
   ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
   ncks -m ~/foo.nc

   20130724: Verified problem still exists
   Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
   Workaround: Use ncrename twice; first rename the variable, then
   rename the dimension.

B. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.

   Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47

C. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

D. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not yet sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

E. NOT YET FIXED
   Report correct chunking and compression information for HDF4 files

   Demonstration:
   ncdump -h -s MOP01-20121231-L1V3.34.10.hdf
   ncks -m MOP01-20121231-L1V3.34.10.hdf

   20131007: Verified problem still exists

   Cause: some libnetCDF library functions fail on HDF4 file inquiries.
   Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults probing
HDF4 file
   Tracking tickets NCF-272, NCF-273

"Sticky" reminders:

A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
   http://nco.sf.net#debian

B. Pre-built Fedora and CentOS RPMs:
   http://nco.sf.net#rpm

C. Pre-built Windows (native) and Cygwin binaries:
   http://nco.sf.net#windows

D. Pre-built AIX binaries:
   http://nco.sf.net#aix

E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
   SWAMP efficiently schedules/executes NCO scripts on remote servers:

   http://swamp.googlecode.com

   SWAMP can work command-line operator analysis scripts besides NCO.
   If you must transfer lots of data from a server to your client
   before you analyze it, then SWAMP will likely speed things up.

F. NCO support for netCDF4 features is tracked at

   http://nco.sf.net/nco.html#nco4

   NCO supports netCDF4 atomic data types, compression, chunking, and
groups.

G. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
   NASA AURA HIRDLS HDF-EOS5
   NASA ICESat GLAS HDF5
   NASA MERRA HDF4
   NASA MODIS HDF4
   NASA SBUV HDF5...

-- 
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(