#92 Copying variable fails with NetCDF4

[Maksym Petrenko]

I start with the following two files:
https://drive.google.com/open?id=0B4xt-T4Lwgs7ME5ycjNkRlIxRm8
https://drive.google.com/open?id=0B4xt-T4Lwgs7Yl9HT05sbFZIcVk

I convert them to nc4 classic:
ncks --7 -O -o 1.nc 1.nc
ncks --7 -O -o 2.nc 2.nc

And then copy a variable from 2.nc into 1.nc:
ncks -A -v AIRX3STD_006_H2OVapMMR_D_H2OPrsLvls_D_1000hPa -o 1.nc 2.nc

This results in

[Segmentation fault (core dumped)]

or

nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): ncoopen()
nco_err_exit(): ERROR Error code is -107. Translation into English with nc_strerror(-107) is "NetCDF: Can't open HDF5 attribute"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

If I replace --7 with --5, everything seems to work properly.

Conf:

NCO netCDF Operators version 4.5.3 built by mockbuild on XXX at Nov 16 2015 14:11:22
ncks version "4.5.3"
Linked to netCDF library version 4.2.1.1, compiled Feb 2 2015 18:08:18
Copyright (C) 1995--2015 Charlie Zender

thanks,
Maksym

[Charlie Zender]

Hi Maxim,
thanks for this report. i can reproduce the problem with the latest NCO, except i receive error (-127) "Bad chunk sizes". Your commands work with -3,-5,-6 and fail with -4, -7, so at least you have some workarounds. Not sure what is going on. Will post again when I know more.
cz

[Maksym Petrenko]

Thank Charlie! We are limited to --3 and --7 since we need support for groups (if I recall the issue correctly). We are also starting to hit max file size limits on NC3, so --3 would not cut for some of the cases either, unfortunately.

[Charlie Zender]

Progress. I think NCO may have a problem with chunked variables in append mode on netCDF4 files.
Until I verify and fix you can try this workaround with netCDF4 files. It succeeds with NCO 4.6.1:

ncks -A --cnk_dmn time,1 -v AIRX3STD_006_H2OVapMMR_D_H2OPrsLvls_D_1000hPa -o 1.nc 2.nc

i.e., manually set the chunk size to something reasonable.

[Maksym Petrenko]

Thanks Charlie! I have implemented a temporary workaround that tries merging with both --7 and --3, but will add your workaround if we encounter a case where this fails.

[Maksym Petrenko]

Charlie, would this be the same problem?
3.nc https://drive.google.com/open?id=0B4xt-T4Lwgs7dFlSRFpNajJZanM
4.nc https://drive.google.com/open?id=0B4xt-T4Lwgs7QkZKVnE3UjdFZDg

ncks -A -v y_TRMM_3A12_7_surfaceRain 3.nc 4.nc
Segmentation fault: 11
(--cnk_dmn time,1 does not help)

[Charlie Zender]

I suspect it's the same problem, though I need to check on a Linux system before being sure. I have in mind a possible fix to the original problem that I will implement ASAP.

[Maksym Petrenko]

I see, thanks.

[Charlie Zender]

Maksym,
The problem with 3.nc and 4.nc may be different than the bug uncovered with 1.nc and 2.nc. Your command above on 3.nc and 4.nc works fine for me on both Linux and Mac with the latest NCO:

zender@glace:~$ ncks -A -v y_TRMM_3A12_7_surfaceRain ~/3.nc ~/4.nc
zender@glace:~$

However, appending the other variable fails similarly to 1.nc, 2.nc above:

zender@glace:~$ ncks -A -v x_TRMM_3A12_7_surfacePrecipitation ~/3.nc ~/4.nc
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_def_var_chunking()
nco_err_exit(): ERROR Error code is -127. Translation into English with nc_strerror(-127) is "NetCDF: Bad chunk sizes."
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

[Charlie Zender]

Adding --cnk_dmn time,1 does work for me, and eliminates the chunking error shown above:

zender@glace:~$ ncks -A -7 --cnk_dmn time,1 -v x_TRMM_3A12_7_surfacePrecipitation ~/3.nc ~/4.nc
zender@glace:~$

It does not work for you. There may be a combination of issues here. A part of the problem may be driven by flaws in the NCO/netCDF/HDF libraries that have been reduced since NCO 4.5.3. Identifying (and fixing, if possible) the underlying cause of the problem with append mode is the only remaining issue before we release NCO 4.6.2.

[Maksym Petrenko]

Right. Our developer tells me that removing a global attributes makes 3&4 work for him (ncatted -h -a ,global,d,, $varFile2 ), but it seems it still does not work on my Mac, so the problem might indeed be different.

[Charlie Zender]

A followup to let you know that this issue is still a problem. It may be a netCDF library problem, or a problem with my understanding of the netCDF library. I've examined the relevant NCO code and cannot find any logical flaws. Here is how I demonstrate the problem on our standard dataset:

       ncks -7 -O -v one_dmn_rec_var ~/nco/data/in.nc ~/foo1.nc
       ncks -7 -O -v one_dmn_rec_wgt ~/nco/data/in.nc ~/foo2.nc
       ncks -A -v one_dmn_rec_wgt ~/foo2.nc ~/foo1.nc # borken
       ncks -A -C -v one_dmn_rec_wgt ~/foo2.nc ~/foo1.nc # borken
       ncks -A --cnk_dmn time,1 -v one_dmn_rec_wgt ~/foo2.nc ~/foo1.nc # works

Any suggestions would be welcome.

[Maksym Petrenko]

Thanks Charlie!

[Charlie Zender]

Maksym,
We recently improved the chunking code and the test commands above now work for me with the latest snapshot, 4.6.8-alpha03. If you have a chance to try this, please do...otherwise will be available when 4.6.8 is released, maybe in 2 weeks.

[Charlie Zender]

This bug was patched in response to the another issue with chunking. Fix will appear in 4.6.8.