#84 Appending variable with ncks messes up values in a different variable

[Bill Sacks]

I have run into this problem with nco versions 4.3.7, 4.4.2 and 4.5.1 on NCAR's yellowstone machine (using the netcdf/4.3.0 module), and with version 4.4.2 on my Mac.

The full module list on yellowstone was:
Currently Loaded Modules:
1) ncarbinlibs/1.1 3) ncarcompilers/1.0 5) netcdf/4.3.0 7) git/2.3.0 9) all-python-libs 11) ncl/6.3.0 13) perlmods/5.0
2) intel/13.1.2 4) ncarenv/1.0 6) subversion/1.8.9 8) python/2.7.7 10) R/2.15.2.opt 12) ncview/2.1.1 14) nco/4.5.1

Steps to reproduce:

(1) Unzip & untar the attached file (reproducer.tar.gz)

(2) cp clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc test.nc

(3) ncks -A -v COL_Z colz.nc test.nc

(4) ncdump -v plant_nalloc test.nc > dump_new

(5) ncdump -v plant_nalloc clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc > dump_old

(6) diff dump_old dump_new

Notice that values that used to be FillValue are now garbage.

[Charlie Zender]

Dear Bill,
Thanks for notifying us about this. I can reproduce the problem and it is a serious bug. Silent data corruption of data that was not associated with the original manipulation! Yikes. Good news is there is an easy workaround: convert the input to any format using ncks:

ncks -O -3 ~/Downloads/clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc ~/test.nc
ncks -O -7 ~/Downloads/clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc ~/test.nc
ncks -O -6 ~/Downloads/clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc ~/test.nc
ncks -O -4 ~/Downloads/clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc ~/test.nc

then run the test again and the problem magically goes away (for me anyway):

ncks -A -v COL_Z ~/Downloads/colz.nc ~/test.nc
ncks --cdl -v plant_nalloc ~/test.nc > ~/dump_new
ncks --cdl -v plant_nalloc ~/Downloads/clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc > ~/dump_old
diff ~/dump_old ~/dump_new

please confirm.
i'm unsure what this means. possibilities include the original file has a small, non-fatal bug in its netCDF encoding that the conversion eliminates but that causes the full script to fail. NCO does not invoke different append routines for different filetypes, so i am skeptical that this is an NCO problem. this may be a one-off issue in which case the workaround is the solution. if you notice a reproducible way of generating files that demonstrate this problem, then we'll have more to go on.
cz

[Bill Sacks]

Hi Charlie,

Thanks a lot for the quick reply, and for the suggested workaround! Yes, this workaround works for me, too. (I just tried '-6' using ncks version 4.4.2 on yellowstone, and that worked.)

In this particular case, the workaround has other undesirable side-effects: namely, dropping some of the unused dimensions from the file (which are needed as metadata)... I have never figured out if there's a way to suppress that feature of ncks. But I found another workaround that does the trick: After doing the processing I needed to do (resulting in file clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151026.nc with messed-up plant_nalloc), replacing plant_nalloc with the correct version, using:

ncrename -v plant_nalloc,plant_nalloc_old clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151026.nc

ncks -A -v plant_nalloc clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151026.nc

The problematic file (with date-stamp 151004) was produced by a tool inside CESM's CLM, and it's hard for me to see what could have caused the malformatting. But I'm able to reproduce the original problem with a number of other files produces by this tool, so it's quite possible that it is the real culprit. I will dig a bit more....

Thank you,
Bill

[Charlie Zender]

Use ncks --rdd
C

[Bill Sacks]

I get:

ncks: unrecognized option '--rdd'

From reading the documentation, though, I don't think that's exactly what I want: I didn't mean that it removes degenerate dimensions from variables, but rather that it removes information about a given dimension from the file entirely:

ncks -O -6 clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc test3.nc
ncdump -h clmi.ICRUCLM45BGCCROP.78pfts.levis.10x15_USGS_simyr2000_c151004.nc > hdr_old
ncdump -h test3.nc > hdr_new
sort hdr_old > hdr_old.s
sort hdr_new > hdr_new.s
diff hdr_old.s hdr_new.s

[some irrelevant diffs, then...]
3376d3377
< cohort = 22240 ;
3801d3801
< levurb = 5 ;
3804d3803
< string_length = 64 ;

But sorry, I didn't mean for this bug report to go off on this tangent.

Thank you,
Bill

[Charlie Zender]

My mistake, I meant use ncks --rad not --rdd.
c

[Bill Sacks]

Perfect - that's exactly what I need!

Thank you for this! -- and for all you have put into NCO: I don't know how I'd live without it!

[Bill Sacks]

I have dug into this problem a little more. I find that it appears reproducibly with restart files written by CESM's CLM. However, it only appears when files are written with pnetcdf (not with standard netcdf). Our best guess at this point is that this is actually a pnetcdf bug. Charlie, I have cc'ed you on a bug report that I just sent to the pnetcdf team.

Thank you,
Bill

[Bill Sacks]

Wei-keng Liao on the pnetcdf team has tracked this down to a bug in the netcdf library, which affects files written by pnetcdf. I have submitted this bug to the netcdf team.

Sorry for implicating NCO!

[Charlie Zender]

As the discussion shows, this was due to a pnetcdf bug fixed nearly a year ago.