[nco-announce] netCDF operators NCO version 4.2.0 are ready
Command-line operators for netCDF and HDF files
Brought to you by:
zender
Menu
▾
▴
[nco-announce] netCDF operators NCO version 4.2.0 are ready
|
From: Charlie Z. <ze...@uc...> - 2012-06-12 18:57:09
|
The netCDF operators NCO version 4.2.0 are ready. http://nco.sf.net (Homepage) http://dust.ess.uci.edu/nco (Homepage "mirror") This release is notable for its improved support for "bounds" and "coordinates" attributes (often used in the CMIP5 archive), and for its native support for M$ Windows. The minor release change in numbering of this release, from 4.1.0 to 4.2.0, was made to mark the change in NCO status to funded (by NASA) status. It also helps to alleviate some confusion between netCDF and NCO version numbering. netCDF 4.2.x just came out and it seems less confusing to call the NCO developed with it NCO 4.2.x rather than NCO 4.1.x. Work on NCO 4.2.1 is underway, focused on making a more robust build/configuration environment for RHEL, CentOS, and AIX, as well as addressing the MM3 issue in other (non-ncks) operators, and on simplifying issues described in the KNOWN BUGS NOT YET FIXED section below. Enjoy, Charlie Other "New stuff" in 4.2.0 summary (full details always in ChangeLog): A. Operators now treat variables specified in "coordinates" and "bounds" attributes more consistently. B. Native, self-extracting binary executables for Microsoft Windows are now distributed from the NCO homepage. We hope to keep these exectuables up-to-date. No compilation necessary. No UNIX necessary. Your mother could use these. However, they are not yet thoroughly tested. Please send us feedback. http://nco.sf.net#windows C. ncap2 append mode (i.e., ncap2 -A) now works when variables to be appended contain dimensions that already exist in the output file. (The rest of NCO has always done this correctly, but ncap2 is architected quite differently). Thanks to user mullers for reminding us about this bug, and to Henry Butowsky for fixing it. D. The NCO documentation contains examples of shell commands defined by post-processing the verbose output from ncks. Currently these commands include nclist and ncattget, that produce, respectively, uncluttered lists of the variables in a file and of the value of an attribute. These examples should help users tailor NCO output to their own needs: http://nco.sf.net/nco.html#nclist http://nco.sf.net/nco.html#ncattget E. ncecat no longer glues a record dimension to any auxiliary coordinate variables (latixy, longxy, ...) or bounds (lat_bnds, lon_bnds, ...). These are variables specified in the "coordinates" or "bounds" attribute, respectively. Note that regular (1D) coordinate variables (lat, lon, time) may also be listed as auxiliary coordinate variables. But ncecat never added a record dimension to "normal" 1D coordinates. Thanks to Patrick Brockmann for this suggestion. F. Microsoft Windows support. NCO source code now contains all modifications required to build under Visual Studio 2010. The necessary changes are generally indicated by the presence of the _MSC_VER pre-processor token. Compilation is only supported with the C++ compiler (since M$ does not support C99). Qt Project environment settings are provided in the nco/qt directory. http://nco.sf.net/nco.html#windows G. All operators allow specification of buffer size for reading and writing files. Specify size with --bfr_sz_hnt or --buffer_size_hint: # Request 2 MB file buffer instead of default 8 kB buffer ncks -O -D 3 --bfr_sz=2097152 ~/nco/data/in.nc ~/foo.nc Larger sizes can increase access speed by reducing the number of system calls netCDF makes to read/write data from/to disk. Thanks to Andy Mai for reporting Rob Latham's suggestion for this. http://nco.sf.net/nco.html#bfr H. NaN means Not a Number in IEEE-speak Recently I learned that some people use NaN for a missing value, and that netCDF supports NaNs. So NCO should, too, but did not. Currently the workaround is to change the missing value to non-NaN, then perform arithmetic, then change the missing value back to Nan. However, show of hands: How many people would like NCO to handle missing values that are NaN without using a workaround? i.e., for NCO to do all the checking itself? ncwa now correctly handles averages of missing values that are NaN. Should we spend time supporting NaN for all arithmetic functions? http://nco.sf.net/nco.html#nan I. ncks now prints variables of type NC_INT with %i instead of %li formatting. This fixes TODO nco1040. J. ncra properly initializes counting and accumlation arrays. This could cause a bug with the ncra -y ttl (i.e., total) function. This fixes TODO nco1039 and nco1051. Thanks to Ken5746 for reporting this. K. Fix double-free() error triggered by misordering UDUnits2 free(). Problem caused ncrcat to fail when concatenating record dimensions with non-time units. Thanks to Andrew Wittenburg and to Peter Campbell for reporting this. KNOWN BUGS NOT YET FIXED: This section of the ANNOUNCE file reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.2.0 built/tested with netCDF 4.2-snapshot2012040523 on top of HDF5 hdf5-1.8.8 with these methods: cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or- cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism A. NOT YET FIXED Correctly read netCDF4 input over DAP, write netCDF4 output, then read resulting file. Replacing netCDF4 with netCDF3 in either location of preceding sentence leads to success. DAP non-transparency: Works locally, fails through DAP server. Demonstration: ncks -4 -O -v three_dmn_rec_var http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name" 20120611: Verified problem still exists Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions B. NOT YET FIXED netCDF4 library fails when renaming dimension and variable using that dimension, in either order. Works fine with netCDF3. Problem with netCDF4 library implementation. Demonstration: ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc # Breaks with "NetCDF: HDF error" ncks -m ~/foo.nc 20120611: Verified problem still exists Bug report filed: netCDF #YQN-334036: problem renaming dimension and coordinate in netCDF4 file C. NOT YET FIXED Unable to retrieve contents of variables including period '.' in name Metadata is returned successfully, data is not. DAP non-transparency: Works locally, fails through DAP server. Demonstration: ncks -O -C -D 3 -v var_nm.dot -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to find variable 20120611: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts. NB: Problem hard to fix since DAP interprets '.' as structure delimiter in query string of HTTP requests. Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47 "Sticky" reminders: A. Pre-built, up-to-date Debian Sid & Ubuntu packages: http://nco.sf.net#debian B. Pre-built Fedora and CentOS RPMs: http://nco.sf.net#rpm C. Pre-built Windows (native) and Cygwin binaries: http://nco.sf.net#windows D. Pre-built AIX binaries: http://nco.sf.net#aix E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)? SWAMP efficiently schedules/executes NCO scripts on remote servers: http://swamp.googlecode.com SWAMP can work command-line operator analysis scripts besides NCO. If you must transfer lots of data from a server to your client before you analyze it, then SWAMP will likely speed things up. F. NCO support for netCDF4 features is tracked at http://nco.sf.net/nco.html#nco4 NCO supports netCDF4 atomic data types, compression, and chunking. G. Have you seen the NCO logo candidates by Tony Freeman, Rich Signell, Rob Hetland, and Andrea Cimatoribus? http://nco.sf.net Tell us what you think... -- Charlie Zender, Department of Earth System Science University of California, Irvine 949-891-2429 )'( |