[nco-announce] netCDF Operators NCO version 4.3.9 are ready
Command-line operators for netCDF and HDF files
Brought to you by:
zender
Menu
▾
▴
[nco-announce] netCDF Operators NCO version 4.3.9 are ready
|
From: Charlie Z. <ze...@uc...> - 2013-12-03 00:39:21
|
The netCDF Operators NCO version 4.3.9 are ready. http://nco.sf.net (Homepage) http://dust.ess.uci.edu/nco (Homepage "mirror") We are pleased to report the birth of the operator nces. nces is for netCDF Ensemble Statistics. nces includes the existing functionality of ncea and extends it by allowing groups to be treated as an unlimited dimension of datasets. nces is really ncea with a powerful new feature and a new name. I will debut nces at AGU (Friday 12/13 11:35 AM Moscone West 2010). See below for more info on the feature. Regarding the name... We renamed ncea as nces to more accurately describe the operator. For historical reasons, NCO is full of misnamed operators: NCO itself is misnamed because the operators now work well on HDF (4 and 5) files, not just netCDF files. We tried to change their name to SDO (Scientific Data Operators) but the new name didn't stick. And the "averagers" (ncea, ncra, ncwa) are all misnamed because they all do many statistics (min/max/ttl/rms...) besides averages. This feature addition is an opportune time to rename ncea as nces. ncea is now deprecated, though will remain supported for some time. We faced the same problem of back-compatibility when we renamed ncdiff to ncbo many years ago. We will use the same method. Invoking 'nces' produces the same results as the old 'ncea' so scripts do not break. Under the hood, 'ncea' invokes 'nces --nsm_fl', just as 'ncdiff' invokes 'ncbo --op_typ=subtract'. To cause nces to utilize the new group ensemble mode, invoke it with 'nces --nsm_grp'. This is similar to how ncecat has two modes, record aggregation (the default) and group aggregation, invoked with 'ncecat --rag' and 'ncecat --gag', respectively. This version also improves ncks support for XML (NcML) output. All output now validates without errors with the NcML 2.2 schema. Happy NcML'ing! There is also a bugfix for NaNs in ncatted. Work on NCO 4.4.0 is underway. It focuses on stability and speed. There will be some netCDF4 mop-up, including netCDF4 support for various options (all -X, ncwa -w and -m), and, possibly, improved support for HDF4 files. Heads-up: Unless users report unforeseen difficulties transitioning from ncea to nces, NCO 4.4.0 will rename ncra as ncrs and ncwa as ncws (Record Statistics and Weighted Statistics, respectively). Enjoy, Charlie "New stuff" in 4.3.9 summary (full details always in ChangeLog): NEW FEATURES: A. ncea is now nces or nces --nsm_fl, meaning ensembles of files. Use nces --nsm_grp, for ensembles of groups, a new feature. The same input is valid in both modes. Members of a group ensemble are groups that share the same structure, parent group, and nesting level. Their contents usually have different values because they are realizations of replicated experiments. In group-ensemble mode nces computes the statistics across the ensemble of groups, which may span multiple input files. Files may contain members of multiple ensembles. However, all ensembles must have at least one member in the first input file. Group ensembles behave as a pseudo-record dimension, in that a group ensemble may comprise an arbitrary and extensible number of realizations in each file, and may be composed of multiple files. Output statistics are stored in the parent group by default. If the ensemble members are /cesm/01 and /cesm/02, then the computed statistic will be in /cesm in the output file. The --nsm_sfx option instructs nces to instead store output in a new child group of the parent created by attaching the suffix to the parent group's name, i.e., /cesm/cesm_avg: nces mdl1.nc mdl2.nc mdl3.nc out.nc # Same as ncea nces --nsm_fl mdl1.nc mdl2.nc mdl3.nc out.nc # Same as ncea nces --nsm_grp mdl1.nc mdl2.nc mdl3.nc out.nc nces --nsm_grp --nsm_sfx='_avg' mdl1.nc mdl2.nc mdl3.nc out.nc http://nco.sf.net/nco.html#nces B. Much improved XML/NcML standards compliance and flexibility. Separators for XML now user-configurable with --xml_spr_chr and --xml_spr_nmr for character types (i.e., strings) and numeric types (i.e., numbers), respectively: ncks --xml_spr_chr='***' in.nc # Override default separator '*' ncks --xml_spr_nmr=', ' in.nc # Override default separator ' ' ncks dumps all netCDF3/netCDF4 groups and atomic types (as usual with NCO, netCDF4 non-atomic types are ignored). Output validates without errors against NcML 2.2 schema. Java/NcML users please test and and give us feedback! http://nco.sf.net/nco.html#xml BUG FIXES: A. Fix ncatted so it does not treat zero as NaN when changing a missing value (_FillValue) from NaN to something else. Problem reported by Jianfu Pan. B. Correct XML-mode issues, many reported by Aleksandar Jelenak 1. Print </variable> in metadata mode 2. Adds type="typ_nm" attribute to non-string attribute elements 3. Add separator="foo" element when separator is not whitespace 4. Fix int64-type printing 5. Use NcML not CDL type names 6. Add hidden _Unsigned attribute for unsigned types 7. Add hidden _FillValue attribute for unsigned types 8. Do not print underscores for missing data in XML mode 9. Add shape="" to scalar variables (cough, cough) KNOWN ISSUES NOT YET FIXED: This section of ANNOUNCE reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.3.7 built/tested with netCDF 4.3.1-rc4 snapshot 20131106 on top of HDF5 hdf5-1.8.9 with these methods: cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or- cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism A. NOT YET FIXED netCDF4 library fails when renaming dimension and variable using that dimension, in either order. Works fine with netCDF3. Hence coordinate renaming does not work with netCDF4 files. Problem with netCDF4 library implementation. Demonstration: ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc # Breaks with "NetCDF: HDF error" ncks -m ~/foo.nc 20130724: Verified problem still exists Bug report filed: netCDF #YQN-334036: problem renaming dimension and coordinate in netCDF4 file Workaround: Use ncrename twice; first rename the variable, then rename the dimension. B. NOT YET FIXED (would require DAP protocol change?) Unable to retrieve contents of variables including period '.' in name Periods are legal characters in netCDF variable names. Metadata are returned successfully, data are not. DAP non-transparency: Works locally, fails through DAP server. Demonstration: ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable 20130724: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts. NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string. Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47 C. NOT YET FIXED (would require DAP protocol change) Correctly read scalar characters over DAP. DAP non-transparency: Works locally, fails through DAP server. Problem, IMHO, is with DAP definition/protocol Demonstration: ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc 20120801: Verified problem still exists Bug report not filed Cause: DAP translates scalar characters into 64-element (this dimension is user-configurable, but still...), NUL-terminated strings so MD5 agreement fails D. NOT YET FIXED (NCO problem) Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments Demonstration: ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc 20130724: Verified problem still exists TODO nco1102 Cause: NCO parsing of ncatted arguments is not yet sophisticated enough to handle arrays of NC_STRINGS with embedded delimiters. E. NOT YET FIXED Report correct chunking and compression information for HDF4 files Demonstration: ncdump -h -s MOP01-20121231-L1V3.34.10.hdf ncks -m MOP01-20121231-L1V3.34.10.hdf 20131007: Verified problem still exists Cause: some libnetCDF library functions fail on HDF4 file inquiries. Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults probing HDF4 file Tracking tickets NCF-272, NCF-273 "Sticky" reminders: A. Pre-built, up-to-date Debian Sid & Ubuntu packages: http://nco.sf.net#debian B. Pre-built Fedora and CentOS RPMs: http://nco.sf.net#rpm C. Pre-built Windows (native) and Cygwin binaries: http://nco.sf.net#windows D. Pre-built AIX binaries: http://nco.sf.net#aix E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)? SWAMP efficiently schedules/executes NCO scripts on remote servers: http://swamp.googlecode.com SWAMP can work command-line operator analysis scripts besides NCO. If you must transfer lots of data from a server to your client before you analyze it, then SWAMP will likely speed things up. F. NCO support for netCDF4 features is tracked at http://nco.sf.net/nco.html#nco4 NCO supports netCDF4 atomic data types, compression, chunking, and groups. G. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., NASA AURA HIRDLS HDF-EOS5 NASA ICESat GLAS HDF5 NASA MERRA HDF4 NASA MODIS HDF4 NASA SBUV HDF5... -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'( |