[nco-announce] netCDF Operators NCO version 4.5.4 are ready

[Charlie Z. <ze...@uc...>]

The netCDF Operators NCO version 4.5.4 are ready.

http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)

We are pleased to introduce ncremap, a new operator for regridding.
ncremap is a shell script that wraps ncks to produce a friendly
interface to regridding features. Without any external dependencies,
ncremap applies weights from a pre-exisiting mapfile to a source
data file to produce a regridded dataset. Source and destination
datasets may be on any Curvilinear, Rectangular, or Unstructured
Data (CRUD) grid. When necessary ncremap calls ESMF_RegridWeightGen
or TempestRemap to generate weights and uses those to regrid. ncremap
hides the complexity of regridding from the user, who can know
next-to-nothing about regridding.

Work on NCO 4.5.5 has commenced and will better support eliciting
latitude/longitude coordinates using the CF "coordinates" convention
and regridding variables whose horizontal dimensions are not the
most-rapidly-varying, and ncremap support for calling TempestRemap.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncremap: A new netCDF operator for regridding.
    ncremap is a shell script that wraps ncks to produce a friendly
    interface to regridding features. Without any external dependencies,
    ncremap applies weights from a pre-exisiting mapfile to input data
    file(s) to produce a regrided dataset(s). Source and destination
    datasets may be on any Curvilinear, Rectangular, or Unstructured
    Data (CRUD) grid. ncremap will also, when necessary, use external
    programs (ESMF's ESMF_RegridWeightGen (ERWG), or TempestRemap's
    GenerateOverlapMesh/GenerateOfflineMap) to generate weights and
    mapfiles and use those to regrid. ERWG is distributed in binary
    format with NCL, which many (most?) users already have on their
    system. Or ERWG and TempestRemap may be installed from source.
    Please try ncremap and send us your feedback and suggestions.
    Examples:
    ncremap -i src.nc -d dst.nc -o out.nc
    ls gcm14*cam*0007*.nc | ncremap -a conserve -M -d dst_1x1.nc -O ~/rgr
    http://dust.ess.uci.edu/smn/pst_nco_agu_201512.pdf # AGU Poster
    http://nco.sf.net/nco.html#ncremap

B. ncks prints XML "_Unsigned" attribute for unsigned attribute types.
    Formerly, ncks did not preserve the signedness of attributes.
    Thanks to Aleksandar Jelenak and Ed Armstrong for this suggestion.
    % ncks -v att_var --xml ~/nco/data/in_4.nc | grep Unsigned
     <attribute name="ubyte_att" type="byte" isUnsigned="true" value="0 
1 2 127 128 254 255 0" />
     <attribute name="ushort_att" type="short" isUnsigned="true" 
value="37" />
     <attribute name="uint64_att" type="long" isUnsigned="true" value="0" />
    http://nco.sf.net/nco.html#xml

C. ncap2 now accepts [] and () syntax interchangeably.
    Expressions which formerly had to use one or the other can now
    use either. Brought to you by Henry.

D. ncap2 now conforms arrays by hyperslabbed dimension size rather
    than by dimension name. This mean, e.g., that hyperslabs from
    one dimension can be used to fill-in other dimensions.
    A prototypical example is determining pressure thickness of layers
    as the difference between interface pressure levels.
    For hybrid coordinate system models like CAM, this is now a
    (relatively) simple two-step command with a mixture of different
    dimensions on the LHS and RHS:
    prs_ntf[time,lat,lon,ilev]=P0*hyai+PS*hybi; // Interface pressures
 
prs_dlt[time,lat,lon,lev]=prs_ntf(:,:,:,1:$ilev.size-1)-prs_ntf(:,:,:,0:$ilev.size-2);
    http://nco.sf.net/nco.html#pdel
    Thanks to Philip Cameron-Smith for suggesting and to Henry Butowsky
    for implementing this new feature.

E. NCO now treats variables named wgt_* as weights and avoids
    performing math on them when possible. E.g., ncbo will preserve
    (and not difference) the variable wgt_1. This is the same behavior
    that NCO uses for coordinates, "gw", and masks (named with msk_*.)
    The idea is that weights are a property of the grid and should not
    be differenced, averaged, etc., unless necesssary.
    http://nco.sf.net/nco.html#prc_xcp

F. ncap2 now accepts "sum" as a synonym for "total" in its methods.
    Additionally, ncap2 has a new function,
    ncap_stats_wvariance(var,wgt) to computed weighted standard
    deviations where missing values may be present.
    Additionally, ncap2 now builds with GSL 2.x. All thanks to Henry.
    http://nco.sf.net/nco.html#ncap2

BUG FIXES:

A. ncks -V in (at least) versions 4.5.1--4.5.3 could segfault.
    NB: This is the "-V" (uppercase) not "-v" (lowercase) option.
    The workaround is not to use -V, solution is to upgrade to 4.5.4.
    Thanks to Patrick Nichols for reporting the problem.

B. After dimension reduction operations, ncra and ncwa excise the
    reduced dimensions from the CF "coordinates" attribute, if any.
    Versions 4.4.9--4.5.3 could, under certain circumstances involving
    multiple dimensions, retain extra whitespace in "coordinates".
    A workaround is to dump (ncks --cdl in.nc > out.cdl) then
    regenerate (ncgen -b out.nc out.cdl) the file.
    The solution is to upgrade to 4.5.4.
    Thanks to Richard Strub for reporting the problem.

KNOWN PROBLEMS DUE TO NCO:

    This section of ANNOUNCE reports and reminds users of the
    existence and severity of known, not yet fixed, problems.
    These problems occur with NCO 4.5.3 built/tested with netCDF
    4.4.0-development (20160101) on top of HDF5 hdf5-1.8.13 with:

    cd ~/nco;./configure # Configure mechanism -or-
    cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism

A. NOT YET FIXED (NCO problem)
    Correctly read arrays of NC_STRING with embedded delimiters in 
ncatted arguments

    Demonstration:
    ncatted -D 5 -O -a 
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc 
~/foo.nc
    ncks -m -C -v att_var ~/foo.nc

    20130724: Verified problem still exists
    TODO nco1102
    Cause: NCO parsing of ncatted arguments is not sophisticated
    enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
    ncra/ncrcat (not ncks) hyperslabbing can fail on variables with 
multiple record dimensions

    Demonstration:
    ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

    20140826: Verified problem still exists
    20140619: Problem reported by rmla
    Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
    Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
    Specifying strided hyperslab on large netCDF4 datasets leads
    to slowdown or failure with recent netCDF versions.

    Demonstration with NCO <= 4.4.5:
    time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
    Demonstration with NCL:
    time ncl < ~/nco/data/ncl.ncl
    20140718: Problem reported by Parker Norton
    20140826: Verified problem still exists
    20140930: Finish NCO workaround for problem
    Cause: Slow algorithm in nc_var_gets()?
    Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
    Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (netCDF4 library bug)
    Simultaneously renaming multiple dimensions in netCDF4 file can 
corrupt output

    Demonstration:
    ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc 
# Completes but file is unreadable
    ncks -v one ~/foo.nc

    20150922: Confirmed problem reported by Isabelle Dast, reported to 
Unidata
    20150924: Unidata confirmed problem
    20151221: Verified problem still exists in netCDF library

    Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
    More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
    Unable to retrieve contents of variables including period '.' in name
    Periods are legal characters in netCDF variable names.
    Metadata are returned successfully, data are not.
    DAP non-transparency: Works locally, fails through DAP server.

    Demonstration:
    ncks -O -C -D 3 -v var_nm.dot -p 
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to 
find variable

    20130724: Verified problem still exists.
    Stopped testing because inclusion of var_nm.dot broke all test scripts.
    NB: Hard to fix since DAP interprets '.' as structure delimiter in 
HTTP query string.

    Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
    Correctly read scalar characters over DAP.
    DAP non-transparency: Works locally, fails through DAP server.
    Problem, IMHO, is with DAP definition/protocol

    Demonstration:
    ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p 
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

    20120801: Verified problem still exists
    Bug report not filed
    Cause: DAP translates scalar characters into 64-element (this
    dimension is user-configurable, but still...), NUL-terminated
    strings so MD5 agreement fails

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
    HDF4: AMSR MERRA MODIS ...
    HDF5: GLAS ICESat Mabel SBUV ...
    HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
    http://nco.sf.net#bnr
-- 
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(