Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#50 ncrcat does not handle inter-file non-monotonicity

closed-works-for-me
None
5
2012-03-31
2011-12-07
Brockmann
No

Hi,

Using nco 4.0.8, I have found a new problem that was not in 4.01
when I want to cat 2 files that have different time origin.

$ ncdump -h f1.nc | grep since
time:units = "days since 1850-01-01 00:00:00" ;
$ ncdump -h f2.nc | grep since
time:units = "days since 2006-01-01 00:00:00" ;
$ ncrcat -D1 -Oh f1.nc f2.nc tyty.nc
ncrcat: INFO Build compiler lacked (or user turned off) OpenMP support. Code will execute with single thread in Uni-Processor (UP) mode.
ncrcat: TIMER Metadata setup and file layout before main loop took 0.00 s
ncrcat: INFO/WARNING Inter-file non-monotonicity. Record coordinate "time" does not monotonically increase between last specified record of previous input file (whose name is not cached locally and thus currently unavailable for printing) and first specified record (i.e., record index = 0) of current input file (f2.nc). This message is often informational only and may usually be safely ignored. It is quite common when joining files with "wrapped" record coordinates, e.g., joining a January file to a December file when the time coordinate is enumerated as day of year. It is also common when joining files which employ a "time=base_time+time_offset" convention. Sometimes, however, this message is a warning which signals that the user has joined files together in a different order than intended and that corrective action should be taken to re-order the input files. Output file tyty.nc will contain these non-monotonic record coordinate values (56924.500000, 15.500000) at record indices 1871, 1872.
ncrcat: INFO Moving tyty.nc.pid10995.ncrcat.tmp to tyty.nc...done
ncrcat: TIMER Wallclock-elapsed time for command is 0.10 s

--> the resulting file has a "time" variable and its bounds "time_bnds" variable not correct
in a sens that ncrcat does not calculate the offset due to the different time origin.

With 4.0.1, it was ok at least for the time variable (not the bounds).
Let me know if the feature can be covered we have to cat a lot
of heterogeneous files with different reference time (since aspect in time axis).
The 4.0.5 release has the same behaviour on this point than the 4.0.8 release.

Patrick

--
LSCE/IPSL, Laboratoire CEA-CNRS-UVSQ
Data Analysis and Visualization Engineer
ICMC - IPSL Climate Modelling Centre
--

Discussion

  • Brockmann
    Brockmann
    2011-12-07

     
    Attachments
  • Brockmann
    Brockmann
    2011-12-07

     
    Attachments
  • Charlie Zender
    Charlie Zender
    2011-12-07

    • assigned_to: nobody --> zender
     
  • Charlie Zender
    Charlie Zender
    2011-12-07

    I cannot reproduce this problem with the files you provided.
    Time re-basing works fine with the latest code (4.0.9 beta).
    Nothing I am aware of has changed this since 4.0.8.
    Are you sure your build includes the UDUnits package?
    Check with ncrcat -r
    zender@roulee:~$ ncrcat -r 2>&1 | grep UDUnits
    UDUnits conversions Yes http://nco.sf.net/nco.html#udunits
    UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
    zender@roulee:~$ ncrcat -D 1 -O f1.nc f2.nc f3.nc
    ncrcat: INFO Allowing OS to dynamically set threads
    ncrcat: INFO System will utilize dynamic threading
    ncrcat: INFO Reducing default thread number from 4 to 2, an operator-dependent "play-nice" number set in nco_openmp_ini()
    ncrcat: INFO omp_set_num_threads() used to set execution environment to spawn teams of 1 threads
    ncrcat: TIMER Metadata setup and file layout before main loop took 0.00 s
    ncrcat: INFO Moving f3.nc.pid17890.ncrcat.tmp to f3.nc...done
    ncrcat: TIMER Wallclock-elapsed time for command is 0.15 s
    zender@roulee:~$ ncks -v time f3.nc | m
    ...visual inspection shows it is monotonic.

     
  • Charlie Zender
    Charlie Zender
    2011-12-07

    • status: open --> pending-works-for-me
     
  • Brockmann
    Brockmann
    2011-12-07

    • status: pending-works-for-me --> open-works-for-me
     
  • Brockmann
    Brockmann
    2011-12-07

    Indeed, I get

    ncrcat -r 2>&1 | grep UDUnits
    UDUnits conversions No http://nco.sf.net/nco.html#udunits
    UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits

    I have to change this.
    I will let you know if it solves the problem. Thanks to point me this

    Patrick

     
  • Brockmann
    Brockmann
    2011-12-08

    Hi,

    Do I have both UDunits and UDUnits2 set at Yes to
    have a right concatenation in my case.
    Could you confirm this because I thought that
    UDUnits2 overruled UDunits ?

    Could you aslo inspect the time boundaries given by the bounds attribut in the result file ?
    Because with a ncrcat at 4.0.8 with both UDUnits and UDUnits2 set at Yes, the time-bnds is not
    processed correctly and is not monotonic as I expect

    Using ferret, you get:
    *** NOTE: Error in bounds "time_bnds" or bounds do not enclose point on axis time
    *** NOTE: Substituting coordinate midpoints

    extract of a ncdump on f1.nc
    double time(time) ;
    time:bounds = "time_bnds" ;
    time:units = "days since 1850-01-01 00:00:00" ;
    time:calendar = "noleap" ;
    time:axis = "T" ;
    time:long_name = "time" ;
    time:standard_name = "time" ;
    double time_bnds(time, bnds) ;

    Patrick

     
  • Charlie Zender
    Charlie Zender
    2011-12-08

    Patrick,
    I _think_ that either UDUnits 1 or UDUnits 2 will accomplish time-rebasing.
    Not 100% sure because I don't build with UDUnits 1 anymore.
    UDUnits 2 definitely should work.
    Also not sure that UDUnits flags displayed by ncrcat -r tell us what we think it is.

    What I am sure of:
    My builds with UDUnits2 perform time re-basing correctly for the record coordinate.
    It sounds like you now have ncrcat correctly re-basing time but not time_bnds.
    Is this true?

    The associated "bounds" variables like time_bnds are, as you may know, designed to adhere to CF metadata conventions. NCO does not currently re-base the bounds coordinates. It's certainly possible (with a lot of coding) to do this, but that code has not been written (this is TODO nco1020).

    What to do to re-base time_bnds: There are many approaches, none of them are elegant. I'm tempted to suggest you simply delete time_bnds, because it's mostly just a repeat of the time information. Assuming you reallly need it, then you might re-base time_bnds (to the same base as in the first input file) in each input file by hand (i.e., with ncap2) before using ncrcat, then ncrcat will re-base your time coordinate and time_bnds will already be done.

    hope this helps.
    cz

     
  • Charlie Zender
    Charlie Zender
    2012-03-31

    • status: open-works-for-me --> closed-works-for-me
     
  • Charlie Zender
    Charlie Zender
    2012-03-31

    Closed due to inactivity and ... it works for me.