Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

nco-4.2.3 regression with HDF5 files

Developers
ocehugo
2013-01-17
2013-10-17
  • ocehugo
    ocehugo
    2013-01-17

    Hello,

    i've found another regression over nco-4.2.3, that was included, apparently only in this version and only occurs with netcdf4/hdf5 files. I've tested it with newest nco-4.2.4 and the command works.

    This version return me the following problem when hyperslabing over record dimensions (`ncks -d time,0,10 file.nc file2.nc`) :

    ncks: ERROR User defined the output record dimension to be "time". Yet in the input variable "tmp2m" the record dimension is dimension number 2. NCO (and the netCDF3 API) only supports the record dimension as the first (i.e., least rapidly varying) dimension. Consider using ncpdq to permute the location of the record dimension in the output file.
    

    But looking at my file, the record dimension is in the right place(first) and obviously i'm working with n4/hdf5 files so this would not be a problem. When converting to netcdf3 and hyperslabing again, the command works.

    PS: The website points to 4.2.3 for stable release ,but according to the changelog the 4.2.4 is the stable now. So anyone updating the code would probably be surprised with this bug, including me :P.

    Sugestion:
    I know that i'm just a little ant over the number of people that contributed to the code/project, but it's not a good time to add a mirror of nco to github!? The bugtracker and code navigation are so simple…even in the web interface this kind of regression would be simple to look at the code diffs …

    Thankz

     
  • Charlie Zender
    Charlie Zender
    2013-01-18

    The current stable release is 4.2.3.
    We have tagged 4.2.4 but it is not released.
    We will continually re-tag until the release is made.
    Yes, it's a little messy. Sorry. That is just the way we roll releases.
    They are not as atomic as they should be and sometimes get spread-out over a few days.
    4.2.4 will be out by next week.
    cz

     
  • Pedro Vicente
    Pedro Vicente
    2013-01-18

    @ocehugo

    SF has a code browse/diff feature for CVS.
    Go to the menu above "Code", then "CVS browse"

    then click the "nco" root group link to show the folder contents

    then you can order the file list by choosing any of

    File Rev. Age Author Last log entry

    then choose a file link

    each file has a list of revisions;

    each revision has its own link
    for example
    Diff to previous 1.361

    here's a direct URL to a file

    http://nco.cvs.sourceforge.net/viewvc/nco/nco/src/nco/nco_grp_utl.c?r1=1.361&r2=1.362&sortby=date

    it shows the differences with the previous version in colors and columns side by side

    Legend:
    Removed from v.1.361
    changed lines
      Added in v.1.362

    Pedro

     
  • ocehugo
    ocehugo
    2013-01-21

    hello again,

    thankz for the info zender, but have you reproduced the problem!?

    @pvicente,
    yes i know that this option should be hide somewhere in SF but it took me some time to find and see what i want even with your help.Let's be honest, it's not a feature that deserves to be used in the "webspace" and it's lacks a lot of features to be at least informative and easy to use, at least in my particular view.

    Bad news:

    i've found another regression, now with 4.2.4 ; ncks -mk_rec_dim not working anymore with both type of files (n3/n4). For v4.2.3 and olders the option works.

    I think that maybe it's a good thing to look at the nco test's to add some extras regression tests. When something occurs 3 times in a roll (in this case, the regressions), something is wrong in the test tree…maybe more particular tests or newest capabilities tests missing in the test tree, and other kind of stuff that haunt every dev :P.

    My nco profile is limited to get,cut,concatenate,join files and basic maths with ncap2 with n4/hdf5, so maybe  there is more regressions for other user profiles (my experience tells me that 4.0.8 version was working very well in production (again,my profile), so i think that maybe this version can be a very good benchmark or rollback version if you need to make new tests/bug tracking).

    Hope that helps

     
  • Charlie Zender
    Charlie Zender
    2013-01-21

    > thankz for the info zender, but have you reproduced the problem!?

    you said it worked with the latest version so i did not look.
    hyperslabbing is tested extensively in the regression tests (NCO_rgr.pl) so i think whatever bit you has been fixed.
    if not please provide a test case.

    -mk_rec_dim was changed (apparently unannounced, sorry) to -mk_rec_dmn.
    that should work fine.  i have re-added -mk_rec_dim as an alternate form

    you are good at finding problems. please consider adding some regression tests to NCO_rgr.pl.
    thanks for your contined input.
    cz

     
  • ocehugo
    ocehugo
    2013-01-22

    hi zender,

    sorry it was a typo…

    ncks --mk_rec_dmn dim in.nc out.nc
    

    it's not working.

    Example:

    $ ncdump -h in.nc | grep "time ="
        time = 720 ;
              $ ncks --mk_rec_dmn time in.nc out.nc          
              $ ncdump -h out.nc | grep "time ="
        time = 720 ;
    

    -mk_rec_dim return a error.

    With 4.2.3 && 4.0.8 the last ncdump return me the unlimited flag as expected.

     
  • Charlie Zender
    Charlie Zender
    2013-01-22

    i cannot reproduce the problem. please make a small in.nc available that demonstrates the problem.
    cz

     
  • ocehugo
    ocehugo
    2013-01-23

    Here we go:

    NCGEN 4.2.1.1:

    $ncgen --version
    ncgen: invalid option -- '-'
    Usage: ncgen [ -b ] [ -c ] [ -f ] [ -k kind ] [ -x ] [-S struct-format] [-M <name> [ -o outfile]  [ file ... ]
    netcdf library version 4.2.1.1 of Dec 17 2012 15:34:02 $
    

    NCKS version:

    $ ncks --version
    NCO netCDF Operators version "4.2.4" last modified 2013/01/21 ...
    ncks version 4.2.4
    

    CDL Example:

    $cat a.cdl 
    netcdf a {
    dimensions:
        time = 2 ;
        z = 1 ;
        lat = 2 ;
        lon = 2 ;
    variables:
        float myvar(time, z, lat, lon) ;
        double z(z) ;
        double lat(lat) ;
            lat:units = "degrees_north" ;
            lat:long_name = "latitude coordinate" ;
            lat:standard_name = "latitude" ;
        double lon(lon) ;
            lon:units = "degrees_east" ;
            lon:long_name = "longitude coordinate" ;
            lon:standard_name = "longitude" ;
        int time(time) ;
            time:long_name = "forecast time" ;
            time:units = "hours since 1970-01-01T00:00:00Z" ;
    data:
    lat = 0.0, 10.0;
    lon = 0.0, 10.0;
    z = 0.0;
    time = 0.0 , 1.0;
    myvar = 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0 ;
    }
    

    now we create the file:

    $ncgen -k 3 a.cdl -o a.nc
    $ ncdump -h a.nc
    netcdf a {
    dimensions:
        time = 2 ;
        z = 1 ;
        lat = 2 ;
        lon = 2 ;
    variables:
        float myvar(time, z, lat, lon) ;
        double z(z) ;
        double lat(lat) ;
            lat:units = "degrees_north" ;
            lat:long_name = "latitude coordinate" ;
            lat:standard_name = "latitude" ;
        double lon(lon) ;
            lon:units = "degrees_east" ;
            lon:long_name = "longitude coordinate" ;
            lon:standard_name = "longitude" ;
        int time(time) ;
            time:long_name = "forecast time" ;
            time:units = "hours since 1970-01-01T00:00:00Z" ;
    }
    

    and try to add record dimensions:

    $ ncks -D9 --mk_rec_dmn time a.nc b.nc
    ncks: INFO nc__open() will request file buffer of default size
    ncks: INFO nc__open() opened file with buffer size = 0 bytes
    ncks: INFO Group / is at level 0
    ncks: INFO nco_xtr_mk() reports following groups match sub-setting and regular expressions:
    ncks: INFO nco_xtr_mk() reports following variables match sub-setting and regular expressions:
    ncks: INFO nco_xtr_mk() final flags of group /:
     flg_dfl? Yes. flg_mch? No. flg_xtr? Yes. flg_gcv? No. flg_ncs? No.
    ncks: INFO nco_xtr_mk() final flags of variable /myvar:
     flg_dfl? Yes. flg_mch? No. flg_xtr? Yes. flg_vfp? No. flg_vsg? No.
    ncks: INFO nco_xtr_mk() final flags of variable /z:
     flg_dfl? Yes. flg_mch? No. flg_xtr? Yes. flg_vfp? No. flg_vsg? No.
    ncks: INFO nco_xtr_mk() final flags of variable /lat:
     flg_dfl? Yes. flg_mch? No. flg_xtr? Yes. flg_vfp? No. flg_vsg? No.
    ncks: INFO nco_xtr_mk() final flags of variable /lon:
     flg_dfl? Yes. flg_mch? No. flg_xtr? Yes. flg_vfp? No. flg_vsg? No.
    ncks: INFO nco_xtr_mk() final flags of variable /time:
     flg_dfl? Yes. flg_mch? No. flg_xtr? Yes. flg_vfp? No. flg_vsg? No.
    ncks: nco_fl_out_open() reports sizeof(pid_t) = 4 bytes, pid = 5500, pid_sng_lng = 5 bytes, strlen(pid_sng) = 4 bytes, fl_out_tmp_lng = 22 bytes, strlen(fl_out_tmp) = 21, fl_out_tmp = b.nc.pid5500.ncks.tmp
    ncks: INFO nco_aed_prc() examining variable Global
    ncks: TIMER Metadata setup and file layout before main loop took    0.01 s
    ncks: INFO Moving b.nc.pid5500.ncks.tmp to b.nc...done
    ncks: TIMER Wallclock-elapsed time for command is    0.01 s
    $ncdump -h b.nc
    netcdf b {
    dimensions:
        lat = 2 ;
        lon = 2 ;
        time = 2 ;
        z = 1 ;
    variables:
        double lat(lat) ;
            lat:units = "degrees_north" ;
            lat:long_name = "latitude coordinate" ;
            lat:standard_name = "latitude" ;
        double lon(lon) ;
            lon:units = "degrees_east" ;
            lon:long_name = "longitude coordinate" ;
            lon:standard_name = "longitude" ;
        float myvar(time, z, lat, lon) ;
        int time(time) ;
            time:long_name = "forecast time" ;
            time:units = "hours since 1970-01-01T00:00:00Z" ;
        double z(z) ;
    // global attributes:
            :history = "Wed Jan 23 18:20:06 2013: ncks -D9 --mk_rec_dmn time a.nc b.nc" ;
            :NCO = "4.2.4" ;
    }
    

    When using ncks 4.2.3 we get this:

    ncks -D9 --mk_rec_dmn time a.nc c.nc
    ncks: INFO nc__open() will request file buffer of default size
    ncks: INFO nc__open() opened file with buffer size = 0 bytes
    ncks: INFO nco4_inq_trv() reports file contains 0 groups comprising 5 variables, 4 dimensions, and 0 global attributes
    ncks: INFO nco4_var_lst_mk() reports following 5 variables matched sub-setting and regular expressions:
     /myvar
     /z
     /lat
     /lon
     /time
    ncks: nco_fl_out_open() reports sizeof(pid_t) = 4 bytes, pid = 16667, pid_sng_lng = 6 bytes, strlen(pid_sng) = 5 bytes, fl_out_tmp_lng = 23 bytes, strlen(fl_out_tmp) = 22, fl_out_tmp = c.nc.pid16667.ncks.tmp
    ncks: INFO nco_aed_prc() examining variable Global
    ncks: TIMER Metadata setup and file layout before main loop took    0.01 s
    lat, lon, myvar, time, z, ncks: INFO Moving c.nc.pid16667.ncks.tmp to c.nc...done
    ncks: TIMER Wallclock-elapsed time for command is    0.01 s
    $ncdump -h c.nc
    netcdf c {
    dimensions:
        lat = 2 ;
        lon = 2 ;
        time = UNLIMITED ; // (2 currently)
        z = 1 ;
    variables:
        double lat(lat) ;
            lat:units = "degrees_north" ;
            lat:long_name = "latitude coordinate" ;
            lat:standard_name = "latitude" ;
        double lon(lon) ;
            lon:units = "degrees_east" ;
            lon:long_name = "longitude coordinate" ;
            lon:standard_name = "longitude" ;
        float myvar(time, z, lat, lon) ;
        int time(time) ;
            time:long_name = "forecast time" ;
            time:units = "hours since 1970-01-01T00:00:00Z" ;
        double z(z) ;
    // global attributes:
            :history = "Wed Jan 23 18:22:25 2013: ncks -D9 --mk_rec_dmn time a.nc c.nc" ;
            :NCO = "4.2.3" ;
    }
    

    I 've tested add "-3" option to NCO between the calls and multiple options of "-k" in ncgen…everything points to 4.2.4 not working.

    PS: take care…ncgen -k3 is "HDF5"  and -k4 is "hdf5-nc3" according to manpage of netcdf ( i dont know the logic and diffs :P).

     
  • Charlie Zender
    Charlie Zender
    2013-01-23

    Thank you. I have reproduce the problem and wil post again when I have committed a fix.
    cz

     
  • Charlie Zender
    Charlie Zender
    2013-01-28

    I have committed a fix to this problem. The fix will be in NCO 4.2.5, to be released shortly. If you have a chance to test the fix before the release it would be appreciated.
    cz

     
  • ocehugo
    ocehugo
    2013-01-29

    i've checkout and it's working like a charm.

    Also your commit have fix another regression not reported with 4.2.4 : ncks when selecting variables ( ncks -v var in.nc out.nc ) was failing to complete…but i didn't test it very much since i've change my version back). Now i'm with 4.2.5 and i'll stay with him for a while…

    thkz