ncks and _FillValue.

[Nobody/Anonymous]

I have the following problem with ncks:

Y.nc is created from

dimensions:
event=UNLIMITED;
altitude = 201;

variables:

float CO(event,altitude);
        CO:units="vmr";
        CO:_FillValue=-999.9f;
        CO:missing_value=-999.9f;

If lat,lon,lev is copied to this file
from X.nc with
'ncks -A -v lat,lon,lev X.nc Y.nc'

then CO is filled with 0.0 !
Is it a bug or a feature ?

[Charlie Zender]

Good question. It depends on what the values of CO
in Y.nc are before ncks sees them
(i.e., after you create Y.nc from the CDL file).
If CO=0.0 initially, then ncks is fine.
If CO=-999.9 initially, then ncks is broken.
If the latter please send all the .nc files along with
the exact command.

Thanks,
Charlie

[Nobody/Anonymous]

Hi Charlie,
  thanks for the great work!

The 'event' (let's call it eventY) in the
original Y.nc in this case is empty,
so it is extended to the length of
1D vectors lon/lat/lev (eventX)
and CO is filled with 0.0, while i have expected
-999.0, i.e. _FillValue.
If eventY in Y.nc has some nonzero
dimension, that is smaller than
eventX, then CO [EventY...EventX,*] is
set to 0.0. Is this the correct behaviour,
or do i misunderstand something ?
I would prefer that 'ncks' respects
my _FillValue :)

[Charlie Zender]

You are confusing me.
Send an ncdump of Y.nc before ncks touches it.
If CO is zero, ncks is fine.
If CO is non-zero, then there's a bug, i.e.,
the result of appending a record
coordinate is undefined for variables in the target
file, when yes, it should be _FillValue.

[Charlie Zender]

Courtesy copy from email:

Hi Charlie,

   i have reduced the problem to this example:

Y.cdl
- ---------------------------------------------
netcdf dummy {
dimensions:
event = UNLIMITED;
variables:
float CO(event);
        CO:units="vmr";
        CO:_FillValue=-999.9f;
        CO:missing_value=-999.9f;
}
- ----------------------------------------------
X.cdl
- ----------------------------------------------
netcdf dummy {
dimensions:
event = UNLIMITED;
variables:
float lon(event);
data:
lon = 90.;
}
- -----------------------------------------------

Then i do
ncgen -b Y.cdl -o /dev/shm/Y.nc
ncgen -b X.cdl -o /dev/shm/X.nc
ncks -A -v lon /dev/shm/X.nc /dev/shm/Y.nc
ncdump /dev/shm/Y.nc
- --------------------------------------------------
netcdf Y {
dimensions:
        event = UNLIMITED ; // (1 currently)
variables:
        float CO(event) ;
                CO:units = "vmr" ;
                CO:_FillValue = -999.9f ;
                CO:missing_value = -999.9f ;
        float lon(event) ;

// global attributes:
                :history = "Thu Jan 15 10:38:09 2004: ncks -A -v lon
                                  /dev/shm/X.nc /dev/shm/Y.nc" ;
data:
CO = 0 ;
lon = 90 ;
}
- --------------------------------------------------

My question is why CO = 0 and not -999.9 ?

Sincerely,
  Oleg.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.3.2 (GNU/Linux)

iD8DBQFABmNW8HQVQETarcMRArrnAJ9dmPjTeX7keyzenSNxGnNSlqmCJgCgrEFV
nG7sBBPSVTvWo8+dtREWotE=
=9Egq
-----END PGP SIGNATURE-----

[Charlie Zender]

Here is what I think is happening.

> My question is why CO = 0 and not -999.9 ?

Because you're in a gray area.
There's no data in CO when ncks opens it.
And you never append any to it.
You just extend the record length.
Since ncks doesn't know what to write in the new space,
it does not write anything.
So the contents of CO before and after appending are
undefined. They happen to be written as zero.
ncks may not be smart enough to do what you want.

It's unclear to me how hard it would be to get
appends to work as you want. Maybe easy, maybe hard.
I'm not sure whether a netCDF file with no data is
considered "legal" in the first place.
If Unidata says it is, then we'll try to support it.
Except that I personally am not interested in solving
this problem so I would just put it on the TODO
list and hope that someone solves it.

If you want a file of unlimited dimension filled
with missing values you might try ncap and LHS casting.
See the top of ../data/nco_tst.pl for an example where
we use ncap to create a huge file filled with 1's from
a small file with just a few dimensions.

Thanks,
Charlie