[nco-announce] netCDF operators NCO version 4.0.8 are ready

[Charlie Z. <ze...@uc...>]

The netCDF operators NCO version 4.0.8 are ready.

http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")

This release implements support for retrieving files from HPSS storage
systems, such as that now used at NCAR. It also supports the CF
"bounds" convention, the CF "noleap" convention, fixes a bug in
ncatted overwrite-mode, fixes printing of NC_[U]BYTE attributes.

4.0.8 also contains a workaround for the NOFILL bug recently found
to be present in all netCDF versions since 1999. NCO 4.0.8 works
around the NOFILL bug whether or not the underlying netCDF library
has been patched to fix the bug. Often users can re-install
application software (e.g., NCO in RPM or .deb format) more quickly
than they can upgrade base system software (e.g., netCDF).
For these reasons, upgrading to NCO 4.0.8 is recommended.

http://nco.sf.net#bug_nofill

Work on NCO 4.0.9 is underway, focused on issues described in the
KNOWN BUGS NOT YET FIXED section below.

Enjoy,
Charlie

Other "New stuff" in 4.0.8 summary (full details always in ChangeLog):

A. Support retrieving input files from High Performance Storage
   Systems (HPSS). NCAR moved from MSS to HPSS in March so NCO 4.0.8
   dropped support for MSS (msrcp, msread, nrnet commands) and now
   supports the 'hsi get' command, transparently to the user, e.g.,

   ncks -l .
/YO/MAMA/LOVES/HPSS/cam-2rn-co-ch3i-st-19950903.cam2.h0.1997-04-29-64800.nc

   More information, and what to do if your site stores the 'hsi'
   executable in a non-standard location, is at

   http://nco.sf.net/nco.html#hpss

B. Workaround netCDF NOFILL bug:
   Abandon NOFILL mode until netCDF 4.1.3 is released.
   This avoids the (rare, but nasty) bug that can silently corrupt
   writes of netCDF3 files. However, this workaround patch slows
   the writing of netCDF3 files. If write speed is important to you
   you can revert the patch by uncommenting all the lines containing
   NC_NOFILL in the NCO source code.

   http://nco.sf.net#bug_nofill

C. Support CF convention for "bounds" attribute for cell boundaries.
   Thanks to Andrew Schurer for this suggestion.
   Now NCO-generated subsets of CF-compliant files with "bounds" will
   include the coordinates specified by the "bounds" attribute, if any.
   Hence the subsets will themselves be CF-compliant.

   http://nco.sf.net/nco.html#bnd

D. Change "no_leap" to "noleap" for CF-compliance in re-basing code.
   Thanks to Patrick Brockman for reporting this.

E. Fix bug in ncatted where using overwrite-mode to create a new
   (rather than replace an old) _FillValue attribute in a netCDF4 file
   failed. Thanks to Kyle Wilcox for reporting this.

F. Fix bug printing NC_[U]BYTE attributes. Attempting this caused a
   core dump when NCO was compiled with -O optimization. Bytes now
   printed with more-or-less the same convention used by ncgen.

KNOWN BUGS NOT YET FIXED:

   This section of the ANNOUNCE file is intended to make clear the
   existence and severity of known, not yet fixed, problems.

A. POSSIBLY FIXED
   Correctly handle stride argument when accessing some files through
some DAP servers.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -F -D 9 -v weasdsfc -d time,100,110,5
http://nomad3.ncep.noaa.gov:9090/dods/reanalyses/reanalysis-2/6hr/flx/flx ~/foo.nc
# Fails as of 20110114, old DAP server?
   ncks -O -F -D 9 -v three_dmn_rec_var -d time,1,10,5
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in.nc ~/foo.nc #
Works as of 20110114

   20110114: Verified problem still exists on NOAA DAP server
   Bug report not filed: Suggest users upgrade DAP servers?

B. POSSIBLY FIXED
   Correctly handle NC_CHAR coordinates (like fl_dmn) on DAP files.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -D 4 -v fl_dmn -O
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in.nc ~/foo.nc

   20110114: Fixed in Unidata DAP server? Need confirmation works elsewhere
   Bug report filed: netCDF #JZN-949284 NC_CHAR coordinate via DAP

C. NOT YET FIXED
   Correctly average NC_CHAR coordinates from DAP files.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncra -O -C -D 3 -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc in.nc
~/foo.nc # Breaks with "NetCDF: DAP server error"

   20110221: Verified problem still exists
   Bug report filed: netCDF #JZN-949284 NC_CHAR coordinate via DAP

D. NOT YET FIXED
   Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
   Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
   ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"

   20110114: Verified problem still exists
   Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions

E. NOT YET FIXED
   netCDF3 files created under certain conditions are corrupt.
   The corruption occurs silently (without warning or error messages).
   The problem has been seen "in the wild" only on filesystems with
   large block sizes (e.g., Lustre), although it may be more widespread.
   It is caused by a netCDF bug that NCO triggers by invoking
   NOFILL mode for faster writes. Hence it is called the NOFILL bug.
   More information at:

   http://nco.sf.net#bug_nofill

   Demonstration:
   ncks -O -d time,603.75,603.875 -v Q,RELHUM \

/glade/scratch/sghosh/zender/cam-2rn-co-ch3i-st-19950903.cam2.h0.1997-04-29-64800.nc
\
   /glade/scratch/zender/cam-2rn-co-ch3i-st-19950903.cam2.h0.1997-05.W.nc
   ncks -C -H -v Q -d time,603.75 -d lev,25 -d lat,0.0 -d lon,180.0 \
   /glade/scratch/zender/cam-2rn-co-ch3i-st-19950903.cam2.h0.1997-05.W.nc

   Workarounds:
   1. Avoid creating netCDF3 files
   2. Create only netCDF4 files (e.g., ncks -4 ...)
   3. Patch NCO to avoid use of NOFILL mode (results in slower writes):
      Simply comment-out all lines containing NC_NOFILL in NCO source code.
      This patch is used by default in NCO 4.0.8, and will workaround
      the NOFILL bug on all versions of netCDF (i.e., 4.1.2 and earlier).
      Hence NCO 4.0.8 will always correctly write netCDF3 files.
      The NOFILL patch included in NCO 4.0.8 will probably be removed
      in NCO 4.0.9, assuming netCDF 4.1.3, which fixes the underlying
      bug, has been released and is stable. At that time the netCDF3
      write speed of NCO will increase to what it once was.
      Executables at NCAR in ~zender/bin/*/nc* on bluefire and /glade
      have already been patched and will not trigger this bug.

   20110422: Verified problem exists on NCAR /glade and bluefire /ptmp
   Bug report filed: netCDF #VRU-236841: Large block size, silent file
corruption

"Sticky" reminders:

A. All operators support netCDF4 chunking options:
   These options can improve performance on large datasets.
   Large file users: Send us suggestions on useful chunking patterns!
   ncks -O -4 --cnk_plc=all in.nc out.nc
   http://nco.sf.net/nco.html#chunking

B. Pre-built, up-to-date Debian Sid & Ubuntu Maverick packages:
   http://nco.sf.net#debian

C. Pre-built Fedora and CentOS RPMs:
   http://nco.sf.net#rpm

D. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
   SWAMP efficiently schedules/executes NCO scripts on remote servers:

   http://swamp.googlecode.com

   SWAMP can work command-line operator analysis scripts besides NCO.
   If you must transfer lots of data from a server to your client
   before you analyze it, then SWAMP will likely speed things up.

E. NCO support for netCDF4 features is tracked at

   http://nco.sf.net/nco.html#nco4

   NCO supports netCDF4 atomic data types, compression, and chunking.
   NCO 4.0.8 with was built and tested with HDF5 hdf5-1.8.5-patch1 and with
   4.1.2-snapshot2011041822.
   NCO may not build with earlier, and should build with later, netCDF4
releases.
   This is particularly true since NCO 4.0.8 takes advantage of an
   internal change to the netCDF nc_def_var_chunking() API in June 2009.

   export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
   cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
   cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism

F. Have you seen the NCO logo candidates by Tony Freeman, Rich
   Signell, Rob Hetland, and Andrea Cimatoribus?
   http://nco.sf.net
   Tell us what you think...

-- 
Charlie Zender, Department of Earth System Science
University of California, Irvine (949) 891-2429 :)