ncap2 different behivor

Help
Si Liu
2012-12-17
2013-10-17
  • Si Liu

    Si Liu - 2012-12-17

    Bill and I got some problem when running ncap2 commands on our machine.
    The problem is the ncap2 changes the missing_value instead of the _FillValue.

    We built different version of NCO on different platforms. Some of them changed the _FillValue but
    some of them  make changes on missing value instead(As shown below).

    Do you have any hints or suggestions what caused this?
    The demo output is attached at the end.

    Thank you for all your help.
    Have a nice weekend.
    Sincerely Si

        ncap2 -version

    NCO netCDF Operators version "4.2.3" last modified 2012/10/16 built Dec 13 2012 on yslogin2 by siliu
    ncap2 version 4.2.3

        ncdump -h ZWT.nc | grep ZWT

    netcdf ZWT {
            float ZWT(time, lndgrid) ;
                    ZWT:long_name = "water table depth (vegetated landunits only)" ;
                    ZWT:units = "m" ;
                    ZWT:cell_methods = "time: mean" ;
                    ZWT:_FillValue = 1.e+36f ;
                    ZWT:missing_value = 1.e+36f ;
                    :history = "Wed Dec 12 10:48:44 2012: ncks -v ZWT clm4054_ne30g16_I2000.clm2.h0.2000-01_c121107.nc /glade/scratch/sacks/ZWT.nc\n",

        ncap2 -s 'testvar=ZWT; testvar.set_miss(-1)' ZWT.nc test.nc

        ncdump -h test.nc | grep testvar

            float testvar(time, lndgrid) ;
                    testvar:missing_value = -1.f ;
                    testvar:_FillValue = 1.e+36f ;
                    testvar:cell_methods = "time: mean" ;
                    testvar:long_name = "water table depth (vegetated landunits only)" ;
                    testvar:units = "m" ;
                    :history = "Fri Dec 14 16:41:21 2012: ncap2 -s testvar=ZWT; testvar.set_miss(-1) ZWT.nc test.nc\n",

     
  • henry Butowsky

    henry Butowsky - 2012-12-17

    HI there,
    I could NOT recreate your problem.
    I tried with nco version 4.2.3  and  a netcdf3 file and then a netcdf4 file. 
    If you could make your file available via ftp then I'll have another crack

    …hmb

     
  • Charlie Zender

    Charlie Zender - 2012-12-18

    Hi Henry, Si, and Bill,

    Something is strange with missing values in ncap2 (next).
    Fortunately there is a workaround (below).

    A test variable, fll_val_mss_val, has both missing_value and
    _FillValue equal to -999:

    zender@roulee:~/nco$ ncks -C -v fll_val_mss_val ~/nco/data/in.nc
    fll_val_mss_val: type NC_FLOAT, 1 dimension, 3 attributes, chunked? no, compressed? no, packed? no
    fll_val_mss_val size (in RAM) is 4*sizeof(NC_FLOAT) = 4*4 = 16 bytes
    fll_val_mss_val dimension 0: lon, size = 4 NC_FLOAT (Coordinate dimension)
    fll_val_mss_val attribute 0: long_name, size = 18 NC_CHAR, value = _FillValue example
    fll_val_mss_val attribute 1: _FillValue, size = 1 NC_FLOAT, value = -999
    fll_val_mss_val attribute 2: missing_value, size = 1 NC_FLOAT, value = -999

    /fll_val_mss_val
    lon=0 fll_val_mss_val=73
    lon=90 fll_val_mss_val=_
    lon=180 fll_val_mss_val=73
    lon=270 fll_val_mss_val=_

    When we use set_miss() to change the missing value, the attribute is
    changed correctly but the corresponding changes are not made to the
    variable values, i.e., the -999's are not changed to -1's:

    zender@roulee:~/nco$ ncap2 -O -v -s 'testvar=fll_val_mss_val;testvar.set_miss(-1)' ~/nco/data/in.nc ~/foo.nc
    zender@roulee:~/nco$ ncap2 -O -v -s 'fll_val_mss_val.set_miss(-1)' ~/nco/data/in.nc ~/foo.nc
    zender@roulee:~/nco$ ncks -C -v testvar ~/foo.nc
    testvar: type NC_FLOAT, 1 dimension, 3 attributes, chunked? no, compressed? no, packed? no
    testvar size (in RAM) is 4*sizeof(NC_FLOAT) = 4*4 = 16 bytes
    testvar dimension 0: lon, size = 4 NC_FLOAT (Coordinate dimension)
    testvar attribute 0: _FillValue, size = 1 NC_FLOAT, value = -1
    testvar attribute 1: long_name, size = 18 NC_CHAR, value = _FillValue example
    testvar attribute 2: missing_value, size = 1 NC_FLOAT, value = -999

    /testvar
    lon=0 testvar=73
    lon=90 testvar=-999
    lon=180 testvar=73
    lon=270 testvar=-999

    (Henry: nco_aed_prc() will automatically change the variable values if
    you call that from within ncap2. Are you doing this?)

    A somewhat separate issue is that this fails completely:

    zender@roulee:~/nco$ ncap2 -O -v -s 'fll_val_mss_val.set_miss(-1)' ~/nco/data/in.nc ~/foo.nc
    ncap2: WARNING utl_cls::fnd(): set_miss unable to find variable: fll_val_mss_val

    One workaround for all this is to use ncatted to change the missing value:

    zender@roulee:~/nco$ ncatted -O -a _FillValue,,o,f,-1 ~/nco/data/in.nc ~/foo.nc
    zender@roulee:~/nco$ ncks -C -v fll_val_mss_val ~/foo.nc
    fll_val_mss_val: type NC_FLOAT, 1 dimension, 3 attributes, chunked? no, compressed? no, packed? no
    fll_val_mss_val size (in RAM) is 4*sizeof(NC_FLOAT) = 4*4 = 16 bytes
    fll_val_mss_val dimension 0: lon, size = 4 NC_FLOAT (Coordinate dimension)
    fll_val_mss_val attribute 0: long_name, size = 18 NC_CHAR, value = _FillValue example
    fll_val_mss_val attribute 1: _FillValue, size = 1 NC_FLOAT, value = -1
    fll_val_mss_val attribute 2: missing_value, size = 1 NC_FLOAT, value = -999

    /fll_val_mss_val
    lon=0 fll_val_mss_val=73
    lon=90 fll_val_mss_val=_
    lon=180 fll_val_mss_val=73
    lon=270 fll_val_mss_val=_

    Hope this helps,
    cz

     
  • henry Butowsky

    henry Butowsky - 2012-12-18

    Hi Charlie & Si,
    set_miss() - changes just the missing value
    change_miss() changes the missing value AND the missing values in the variable
    … hmb

     
  • Charlie Zender

    Charlie Zender - 2012-12-18

    Thanks, Henry. I had forgotten that. Si, please let us know if that does not solve the problem. I have updated the manual to make this more clear http://nco.sf.net/nco.html#set_miss

     
  • Bill Sacks

    Bill Sacks - 2012-12-20

    Hi Charlie and Henry,

    Thanks for your replies.

    Unfortunately, for this build of the netCDF operators, the change_miss() command has the same erroneous behavior as the set_miss() command: it changes missing_value rather than _FillValue

    $ ncdump -h ZWT.nc | grep ZWT
    netcdf ZWT {
    float ZWT(time, lndgrid) ;
    ZWT:long_name = "water table depth (vegetated landunits only)" ;
    ZWT:units = "m" ;
    ZWT:cell_methods = "time: mean" ;
    ZWT:_FillValue = 1.e+36f ;
    ZWT:missing_value = 1.e+36f ;

    $ ncap2 -s 'testvar=ZWT; testvar.change_miss(-1)' ZWT.nc test_changeMiss.nc

    $ ncdump -h test_changeMiss.nc | grep testvar
    float testvar(time, lndgrid) ;
    testvar:missing_value = -1.f ;
    testvar:_FillValue = 1.e+36f ;
    testvar:cell_methods = "time: mean" ;
    testvar:long_name = "water table depth (vegetated landunits only)" ;
    testvar:units = "m" ;

    I also considered using ncatted; one problem with that solution is that I don't know ahead of time whether the variable is single or double precision, so I'm not sure how to add an attribute with the correct type using ncatted.

    To come back to the original post: The confusing thing to me is why ncap2 version 4.2.3 works right on some platforms (e.g., my Mac laptop), but not on others (yellowstone, using the version that Si built).

    Thank you,
    Bill

     
  • Charlie Zender

    Charlie Zender - 2012-12-20

    Then with ncatted specify double precision and it should work in both cases.

    I do not have a yellowstone account. If CISL creates one for me then I will attempt to discover and fix the ncap2 problem there.

    cz

     
  • Bill Sacks

    Bill Sacks - 2012-12-21

    Hi Charlie,

    Thanks for your continued help with this.

    I may be misunderstanding you, but when I specify double precision with ncatted - e.g.:

    ncatted -a _FillValue,testvar,o,d,1.e36 test_ncatted.nc

    I get a double precision attribute, even if the variable is itself single precision. This is a problem for some utilities that check exact equality of the _FillValue and the data, since some large values like this differ between single and double precision representation.

    Is your user name zender? If so, you do have a home directory on yellowstone… I think in general people with bluefire accounts automatically got yellowstone accounts, though I could be wrong.

    Thanks,
    Bill

     
  • Si Liu

    Si Liu - 2012-12-21

    Dear Charlie and Henry,
    Thank you for your help.
    Charlie, you should have both bluefire and mirage access.
    Let us focus on these two machines first, though the versions could be a little old.
    The data file Bill and I worked on is under /glade/scratch/siliu/NCO_Bill.
    You can get access to it from both bluefire and mirage.

    I also put that data file in my dropbox for your quick access here:
    http://dl.dropbox.com/u/74890350/ZWT.nc

    1) On bluefire, ncap2 changes the _FillValue
    bluefire> which ncap2
    /usr/local/apps/nco-4.0.8/bin/ncap2
    2) On Mirage, ncap2 only changes the missing_value
    which ncap2
    mirage>/fs/local/bin/ncap2

    If I read the message from Henry correct, do you think that "set_miss" statement should only change the missing_value
    not the fill_value? Could you make it clear to me which ((1) or (2)) should be the expected behavior of ncap2?

    It looks like the version I built only work on the missing-value, but some other NCO versions changes the _Filling value instead. There should be something we did in a different way.

    I will also check what we can do for you on YS after the holiday season.

    Wish all of you a Merry Christmas and Wonderful New Year.
    Sincerely Si

     
  • Si Liu

    Si Liu - 2013-01-03

    Any update?
    Si

     
  • henry Butowsky

    henry Butowsky - 2013-01-04

    HI Si,
    nco can be programmed to use "missing_value" or "_FillValue".
    This functionality is set in nco/src/nco.h: line 213. (latest release)
    Its default is " _FillValue" . 

    So it is bizarre  that set_miss() /change_miss() is operating on "missing_value" AND NOT "_FillValue"
    I downloaded and checked your ZWT.nc file. set_miss() it  is working correctly for me.

    If you could compare the relevant sections of nco.h on your build with the latest one on the sourceforge  this might reveal some clues as to whats going on. (Or simply post the version of nco.h on your build - look at the top line of the file)

    I'll let Charlie have a nose about your system as he has a login.

    … hmb

     
  • henry Butowsky

    henry Butowsky - 2013-01-04

    si
    sorry thats line 175 of nco.h

     
  • Si Liu

    Si Liu - 2013-01-08

    Hi Henry,

    I attached the nco.h  source code(line 162-190) I used in nco-4.2.3.
    Best wishes,
    Sincerely Si

    #ifdef MAIN_PROGRAM_FILE /* Current file contains main() */

      /* Tokens and variables with scope limited to main.c, and global variables allocated here */

      int prg; /*  Program ID */
      int prg_get(void){return prg;} /*  Program ID */

      char *prg_nm; /*  Program name */
      char *prg_nm_get(void){return prg_nm;} /*  Program name */

      unsigned short dbg_lvl=0; /*  Debugging level */
      unsigned short dbg_lvl_get(void){return dbg_lvl;} /*  Debugging level */

    # ifdef NCO_USE_FILL_VALUE
      char nco_mss_val_sng="_FillValue"; /*  Missing value attribute name */
      char nco_not_mss_val_sng="missing_value"; /*  Not missing value attribute name */
    # else /* !NCO_USE_FILL_VALUE */
      char nco_mss_val_sng="missing_value"; /*  Missing value attribute name */
      char nco_not_mss_val_sng="_FillValue"; /*  Not missing value attribute name */
    # endif /* !NCO_USE_FILL_VALUE */
      char *nco_mss_val_sng_get(void){return nco_mss_val_sng;} /*  Missing value attribute name */
      char *nco_not_mss_val_sng_get(void){return nco_not_mss_val_sng;} /*  Not missing value attribute name */

    #else /* MAIN_PROGRAM_FILE is NOT defined, i.e., current file does not contain main() */

      /* External references to global variables are declared as extern here
         Variables with local file scope in all files except main.c are allocated here */

    #endif /* MAIN_PROGRAM_FILE is NOT defined, i.e., the current file does not contain main() */

     
  • Charlie Zender

    Charlie Zender - 2013-01-08

    Havn't forgotten this. I have been swamped and intend to look into this tomorrow. Thanks for your patience. cz

     
  • henry Butowsky

    henry Butowsky - 2013-01-08

    HI Si,
    The include file seems in order -
    Guess the next step is to take a peek on the server; Do the appropriate libs on the server to do a build ?
    ..Henry

     
  • Bill Sacks

    Bill Sacks - 2013-01-09

    Just looking through that header file… is there any chance that NCO_MSS_VAL_SNG could have been defined at build time? It looks like  defining that CPP token would cause the incorrect behavior we're seeing here.

    Actually, if I'm interpreting the header file right, it looks like even defining -DNCO_MSS_VAL_SNG=_FillValue would (unintuitively) cause the code to use missing_value rather than _FillValue… but I could be wrong about that.

    Bill

     
  • henry Butowsky

    henry Butowsky - 2013-01-09

    HI Si,
    what you are saying makes sense.
    It would be easy enough to check. Just add a print statement in the following line.

    from:
    char *nco_mss_val_sng_get(void){return nco_mss_val_sng;}
    to:
    char *nco_mss_val_sng_get(void){printf("file:nco.h %s\n",nco_mss_val_sng);   return nco_mss_val_sng;}

    … Henry

     
  • Si Liu

    Si Liu - 2013-01-09

    Hi Bill, Henry, and Charles,

    1) I found what caused the problem. In my script, I defined the CPPFLAGS as:
    `CPPFLAGS='-DNCO_MSS_VAL_SNG=missing_value''.

    That is described in the following page: http://nco.sourceforge.net/nco.txt
    If it is not necessary, I can remove it when building NCO on our machine.

    2) I met another problem when building NCO 4.2.3 recently. I got no problem a few weeks ago when I built the same package.
    I do not think it is related to 1) since it does not matter whether I set the CPPFLAGS.
    The error (when make) is attached at the end. This error (different exception specifier) may be related to some math libraries from my understanding? Do you have any suggestions?

    Thanks a lot.
    Sincerely Si

    g++ -DHAVE_CONFIG_H -I. -I../.. -I./ -I../nco  -I/glade/apps/opt/netcdf/4.2/gnu/4.7.2/include -I/glade/apps/opt/hdf5/1.8.9/gnu/4.7.2/include -I/glade/apps/opt/szlib/2.1/gnu/4.7.2/include -I/glade/apps/opt/zlib/1.2.7/gnu/4.7.2/include -I/glade/apps/opt/netcdf/4.2/gnu/4.7.2/include   -I/glade/apps/opt/antlr/2.7.7/gnu/4.7.2/include -I/glade/apps/opt/udunits/2.1.24/gnu/4.7.2/include   -MT Invoke.o -MD -MP -MF .deps/Invoke.Tpo -c -o Invoke.o Invoke.cc
    In file included from ../nco/nco_var_rth.h:27:0,
                     from ../nco/nco_rth_utl.h:40,
                     from ../nco/nco_pck.h:28,
                     from ../nco/nco_ctl.h:38,
                     from ../nco/nco_mmr.h:37,
                     from ../nco/nco_cnf_typ.h:26,
                     from ../nco/nco_att_utl.h:33,
                     from ../nco/libnco.h:22,
                     from ncap2_utl.hh:52,
                     from vtl_cls.hh:11,
                     from fmc_cls.hh:12,
                     from libnco++.hh:21,
                     from Invoke.cc:13:
    ../nco/nco_rth_flt.h:51:20: error: declaration of 'float acosf(float)' has a different exception specifier
    In file included from /glade/apps/el6/include/math.h:94:0,
                     from ncap2_utl.hh:21,
                     from vtl_cls.hh:11,
                     from fmc_cls.hh:12,
                     from libnco++.hh:21,
                     from Invoke.cc:13:
    /glade/apps/el6/include/bits/mathcalls.h:55:1: error: from previous declaration 'float acosf(float) throw ()'

     
  • Charlie Zender

    Charlie Zender - 2013-01-09

    This is good timing. I just logged into bluefire to see you have solved
    the problem yourself :) You're right, the documentation is in error.
    Sorry about that. I have fixed it. It now reads

    …by compiling NCO with the token definition CPPFLAGS='-UNCO_USE_FILL_VALUE'…

    not

    …by compiling NCO with the token definition CPPFLAGS='-DNCO_MSS_VAL_SNG=missing_value'.

    The reason why is described in nco.h. There is breakage I do not understand
    in the TKN2SNG() CPP macro which would otherwise be my preferred solution.
    The documentation referred to the (non-working) preferred solution rather than
    the kludgy workaround. Until that is fixed, users should not alter NCO_MSS_VAL_SNG.

    Onto the new problem. Si, did you change your build procedure on AIX?
    You are apparently building libnco with g++ not gcc.
    While this is fine in theory (we do it all the time on Linux and Windows),
    it is susceptible to non-portable or system header files for ill-defined
    conventions like intrinsic single precision versions of math library functions.

    That's a long way of saying try building with CC=gcc and CXX=g++
    instead of CC=g++ and CXX=g++.

    If that does not work then we may have to add some new pre-processor tokens
    to nco_rth_flt.h (already a bit messy with AIX) to handle being built with
    g++ on AIX.

    As a first guess you could try changing line 55 from

    #if !defined(HPUX) && !defined(__INTEL_COMPILER)

    to

    #if !defined(HPUX) && !defined(__INTEL_COMPILER) && !defined(_AIX)

    Finally, this is how I build NCO on bluefile. It may be helpful to you:

    cd ~/nco/bld;ANTLR='/contrib/antlr-2.7.7/bin/antlr' ANTLR_ROOT='/contrib/antlr-2.7.7' CURL_LIB='/contrib/curl/7.21.2/lib' GSL_INC='/contrib/gsl-1.12/include' GSL_LIB='/contrib/gsl-1.12/lib' HDF5_ROOT='/contrib/hdf5-1.8.7_seq' LDFLAGS='-lnetcdf -lhdf5_hl -lhdf5 -lz' NETCDF_ROOT='/contrib/netcdf/4.1.3_seq' SZ_LIB='/contrib/szip/lib' UDUNITS_LIB='/contrib/zlib/lib' make -jobs=1 OPTS=D SZ=Y allinone ABI=64;cd -

    cz

     
  • Charlie Zender

    Charlie Zender - 2013-01-09

    And, to be clear, there is no reason to set any command line flags if you want _FillValue. It is the default. Only do CPPFLAGS='-UNCO_USE_FILL_VALUE' if you want missing_value.

     
  • Si Liu

    Si Liu - 2013-01-09

    Hi Charlie and Henry,

    I am working on Yellowstone now and I attached the commands I used to build NCO.
    I will work on the problem tomorrow to see what the problems might be.

    Since bluefire will be decommissioned in a few weeks, we may focus only on Yellowstone.
    I would like to create an account for you on our machine. If Henry also needs one, please let me know.

    Thank you for your help.
    Sincerely Si

    export NETCDF_INC=/glade/apps/opt/netcdf/4.2/gnu/4.7.2/include
    export NETCDF_LIB=/glade/apps/opt/netcdf/4.2/gnu/4.7.2/lib
    export NETCDF4_ROOT=/glade/apps/opt/netcdf/4.2/gnu/4.7.2

    export HDF5_LIB_DIR=/glade/apps/opt/hdf5/1.8.9/gnu/4.7.2/lib
    export UDUNITS2_PATH=/glade/apps/opt/udunits/2.1.24/gnu/4.7.2
    export ANTLR_ROOT=/glade/apps/opt/antlr/2.7.7/gnu/4.7.2

    export CC=gcc
    export CXX=g++
    export CFLAGS="-fPIC -I/glade/apps/opt/netcdf/4.2/gnu/4.7.2/include -I/glade/apps/opt/hdf5/1.8.9/gnu/4.7.2/include -I/glade/apps/opt/szlib/2.1/gnu/4.7.2/include -I/glade/apps/opt/zlib/1.2.7/gnu/4.7.2/include  -L/glade/apps/opt/zlib/1.2.7/gnu/4.7.2/lib -lz -L/glade/apps/opt/szlib/2.1/gnu/4.7.2/lib -lsz -L/glade/apps/opt/netcdf/4.2/gnu/4.7.2/lib -lnetcdf -L/glade/apps/opt/hdf5/1.8.9/gnu/4.7.2/lib -lhdf5 -Wl,-rpath,/glade/apps/opt/netcdf/4.2/gnu/4.7.2/lib -Wl,-rpath,/glade/apps/opt/hdf5/1.8.9/gnu/4.7.2/lib -Wl,-rpath,/glade/apps/opt/szlib/2.1/gnu/4.7.2/lib -Wl,-rpath,/glade/apps/opt/zlib/1.2.7/gnu/4.7.2/lib"
    export CXXFLAGS=$CFLAG
    ##export CPPFLAGS="-DNCO_MSS_VAL_SNG=missing_value"

    ./configure -prefix=/glade/apps/opt/nco/4.2.3/gnu/4.7.2/  -enable-netcdf4 -enable-ncap2 -enable-udunits2

     
  • Si Liu

    Si Liu - 2013-01-17

    Hi Charlie and Henry,

    I have rebuilt NCO without problem and it works as we expected(Changing Fill_Value instead of missing value).
    Thank you for all your suggestions and help.

    I have not heard anything about the account information of Henry so far.
    Please let me know if Henry needs a Yellowstone account ASAP.

    Thanks.
    Sincerely Si

     
  • henry Butowsky

    henry Butowsky - 2013-01-18

    HI Si,
    please communicate thou my email
    henryb@hush.com
    … Henry

     
  • Charlie Zender

    Charlie Zender - 2013-01-18

    Hi Si,
    If Pedro Vicente <pvicente@uci.edu> had a Yellowstone account then
    we would devote some time to getting ./configure working with AIX.
    cz

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks