Regression ncra - 4.2.1 forward...

Developers
ocehugo
2012-11-26
2013-10-17
  • ocehugo
    ocehugo
    2012-11-26

    Hello,

    i've found a regression in ncra, probably associated with the rebase modification. The operation that works with 4.2.0 and udunits do not work anymore with 4.2.1. When averaging over a period and a lot of files, if one file do not have the date inside the hyperslab , the returned unlimited dimension in the output file is zero and no data is saved on the file. I can say that when ncra ignore some file (in the beggining or at the end of hyperslab) the problem show up.

    Sample files (2 sequential time variables from 1988-08-26 00z to 1988-09-31 00z with 8hr stencil).

    Request types:

    ncra -d time,"1988-08-26 00:00:00","1988-08-26 00:00:00" 1.nc 2.nc xx-4.2.1.nc

    Expected (4.2.0): one field centered on the date (no average).
    Returned: nothing - zero zize time dims and only metadata of variables

    ncra -d time,"1988-08-01 00:00:00","1988-08-26 00:00:00" 1.nc 2.nc xx-4.2.1.nc
    Expected (4.2.0): one field at 1988-08-26 (since we only have field at the end date) and ignoring 2.nc
    Returned: nothing - zero zize time dims and only metadata of variables

    When not using the "ignored file" everything runs smoothly.

    ncra -d time,"1988-08-01 00:00:00","1988-08-26 00:00:00" 1.nc xx-4.2.1.nc
    Expected (4.2.0): one field at 1988-08-26 (since we only have field at the end date)
    Returned: one field at 1988-08-26 (since we only have field at the end date)

    ncra -d time,"1988-08-26 00:00:00","1988-08-27 00:00:00" 1.nc 2.nc xx-4.2.1.nc
    Expected (4.2.0): one field centered at  1988-08-26 12:00:00  ignoring file 2.nc
    Returned: nothing - zero zize time dims and only metadata of variables

    When the slab contains the range date inside all files all happens as expected.

    ncra -d time,"1988-08-26 00:00:00","1988-08-30 00:00:00" 1.nc 2.nc xx-4.2.1.nc
    Expected: one field with average center on 1988-08-28
    Returned: one field with average center on 1988-08-28

    Very easy to reproduce and breaking a very nice feature of ncra… ncra -v pressure -d time,"2000-01-01 00:00:00","2001-01-01 00:00:00" *pressure_fields*.nc 2000_pmean.nc (if any file inside the *pressure_fields*.nc  list have a date not inside the hyperslab, we cant get the mean field).

    hope that helps and going back to 4.2.0 hehe

     
  • Charlie Zender
    Charlie Zender
    2012-11-26

    Thank you for reporting this regression.

    I have reproduced the problem with this simplified code

          ncap2 -O -v -s 'one_dmn_rec_var=one_dmn_rec_var' ~/nco/data/in.nc ~/in1.nc
          ncap2 -O -s 'time=time+10' ~/in1.nc ~/in2.nc
          ncra  -O -d time,"1964-03-12 00:00:00","1964-03-23 00:00:00" -p ~ in1.nc in2.nc ~/foo.nc

    The results are zeros in both fields, as in your posting.
    One workaround is to use index-based hyperslabbing.

    I will post a followup here when I've  committed a fix.

    cz

     
  • Charlie Zender
    Charlie Zender
    2012-12-17

    Hi again,
    I just committed a fix to this problem.
    Would appreciate if you could test it.
    Here is the description from the upcoming ANNOUNCE:

    A. Fix bug introduced in 4.2.1 where ncra did not write results of
       final normalization if trailing file(s) was/were superfluous.
       In the most common case, all zeros appeared in the output file.
       http://nco.sf.net#bug_ncra_flg_bfr_nrm

    FYI the test case for this problem ultimately reduced to

          ncks  -O -C -d time,0 -v time ~/nco/data/in.nc ~/in1.nc
          ncks  -O -C -d time,1 -v time ~/nco/data/in.nc ~/in2.nc
          ncra  -O -d time,0 -p ~ in1.nc in2.nc ~/foo.nc # borken

    cz