#50 ncrcat does not handle inter-file non-monotonicity



Using nco 4.0.8, I have found a new problem that was not in 4.01
when I want to cat 2 files that have different time origin.

$ ncdump -h f1.nc | grep since
time:units = "days since 1850-01-01 00:00:00" ;
$ ncdump -h f2.nc | grep since
time:units = "days since 2006-01-01 00:00:00" ;
$ ncrcat -D1 -Oh f1.nc f2.nc tyty.nc
ncrcat: INFO Build compiler lacked (or user turned off) OpenMP support. Code will execute with single thread in Uni-Processor (UP) mode.
ncrcat: TIMER Metadata setup and file layout before main loop took 0.00 s
ncrcat: INFO/WARNING Inter-file non-monotonicity. Record coordinate "time" does not monotonically increase between last specified record of previous input file (whose name is not cached locally and thus currently unavailable for printing) and first specified record (i.e., record index = 0) of current input file (f2.nc). This message is often informational only and may usually be safely ignored. It is quite common when joining files with "wrapped" record coordinates, e.g., joining a January file to a December file when the time coordinate is enumerated as day of year. It is also common when joining files which employ a "time=base_time+time_offset" convention. Sometimes, however, this message is a warning which signals that the user has joined files together in a different order than intended and that corrective action should be taken to re-order the input files. Output file tyty.nc will contain these non-monotonic record coordinate values (56924.500000, 15.500000) at record indices 1871, 1872.
ncrcat: INFO Moving tyty.nc.pid10995.ncrcat.tmp to tyty.nc...done
ncrcat: TIMER Wallclock-elapsed time for command is 0.10 s

--> the resulting file has a "time" variable and its bounds "time_bnds" variable not correct
in a sens that ncrcat does not calculate the offset due to the different time origin.

With 4.0.1, it was ok at least for the time variable (not the bounds).
Let me know if the feature can be covered we have to cat a lot
of heterogeneous files with different reference time (since aspect in time axis).
The 4.0.5 release has the same behaviour on this point than the 4.0.8 release.


Data Analysis and Visualization Engineer
ICMC - IPSL Climate Modelling Centre


  • Brockmann

    Brockmann - 2011-12-07
  • Brockmann

    Brockmann - 2011-12-07
  • Charlie Zender

    Charlie Zender - 2011-12-07
    • assigned_to: nobody --> zender
  • Charlie Zender

    Charlie Zender - 2011-12-07

    I cannot reproduce this problem with the files you provided.
    Time re-basing works fine with the latest code (4.0.9 beta).
    Nothing I am aware of has changed this since 4.0.8.
    Are you sure your build includes the UDUnits package?
    Check with ncrcat -r
    zender@roulee:~$ ncrcat -r 2>&1 | grep UDUnits
    UDUnits conversions Yes http://nco.sf.net/nco.html#udunits
    UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
    zender@roulee:~$ ncrcat -D 1 -O f1.nc f2.nc f3.nc
    ncrcat: INFO Allowing OS to dynamically set threads
    ncrcat: INFO System will utilize dynamic threading
    ncrcat: INFO Reducing default thread number from 4 to 2, an operator-dependent "play-nice" number set in nco_openmp_ini()
    ncrcat: INFO omp_set_num_threads() used to set execution environment to spawn teams of 1 threads
    ncrcat: TIMER Metadata setup and file layout before main loop took 0.00 s
    ncrcat: INFO Moving f3.nc.pid17890.ncrcat.tmp to f3.nc...done
    ncrcat: TIMER Wallclock-elapsed time for command is 0.15 s
    zender@roulee:~$ ncks -v time f3.nc | m
    ...visual inspection shows it is monotonic.

  • Charlie Zender

    Charlie Zender - 2011-12-07
    • status: open --> pending-works-for-me
  • Brockmann

    Brockmann - 2011-12-07
    • status: pending-works-for-me --> open-works-for-me
  • Brockmann

    Brockmann - 2011-12-07

    Indeed, I get

    ncrcat -r 2>&1 | grep UDUnits
    UDUnits conversions No http://nco.sf.net/nco.html#udunits
    UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits

    I have to change this.
    I will let you know if it solves the problem. Thanks to point me this


  • Brockmann

    Brockmann - 2011-12-08


    Do I have both UDunits and UDUnits2 set at Yes to
    have a right concatenation in my case.
    Could you confirm this because I thought that
    UDUnits2 overruled UDunits ?

    Could you aslo inspect the time boundaries given by the bounds attribut in the result file ?
    Because with a ncrcat at 4.0.8 with both UDUnits and UDUnits2 set at Yes, the time-bnds is not
    processed correctly and is not monotonic as I expect

    Using ferret, you get:
    *** NOTE: Error in bounds "time_bnds" or bounds do not enclose point on axis time
    *** NOTE: Substituting coordinate midpoints

    extract of a ncdump on f1.nc
    double time(time) ;
    time:bounds = "time_bnds" ;
    time:units = "days since 1850-01-01 00:00:00" ;
    time:calendar = "noleap" ;
    time:axis = "T" ;
    time:long_name = "time" ;
    time:standard_name = "time" ;
    double time_bnds(time, bnds) ;


  • Charlie Zender

    Charlie Zender - 2011-12-08

    I _think_ that either UDUnits 1 or UDUnits 2 will accomplish time-rebasing.
    Not 100% sure because I don't build with UDUnits 1 anymore.
    UDUnits 2 definitely should work.
    Also not sure that UDUnits flags displayed by ncrcat -r tell us what we think it is.

    What I am sure of:
    My builds with UDUnits2 perform time re-basing correctly for the record coordinate.
    It sounds like you now have ncrcat correctly re-basing time but not time_bnds.
    Is this true?

    The associated "bounds" variables like time_bnds are, as you may know, designed to adhere to CF metadata conventions. NCO does not currently re-base the bounds coordinates. It's certainly possible (with a lot of coding) to do this, but that code has not been written (this is TODO nco1020).

    What to do to re-base time_bnds: There are many approaches, none of them are elegant. I'm tempted to suggest you simply delete time_bnds, because it's mostly just a repeat of the time information. Assuming you reallly need it, then you might re-base time_bnds (to the same base as in the first input file) in each input file by hand (i.e., with ncap2) before using ncrcat, then ncrcat will re-base your time coordinate and time_bnds will already be done.

    hope this helps.

  • Charlie Zender

    Charlie Zender - 2012-03-31
    • status: open-works-for-me --> closed-works-for-me
  • Charlie Zender

    Charlie Zender - 2012-03-31

    Closed due to inactivity and ... it works for me.


Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.

No, thanks