Adding a record dimension to a variable

[Todd Mitchell]

It is common to get observational files with no record dimension and with variables defined like SST( lat, lon). 
The NCO method to define time as the record dimension is well established, but in order to
concatenate the files you have to redefine SST(lat,lon) as SST(time,lat,lon).  How do you do this?

[henry Butowsky]

Hi Todd,
you can do this using the 'ncecat' operator. See the manual for more details.
http://nco.sourceforge.net/nco.html#ncecat-netCDF-Ensemble-Concatenator

…hb

[Todd Mitchell]

Hi HB,

Thank you for your reply.  I can see now that my problem is more complicated.  My file is from
ftp://ftp.ifremer.fr/ifremer/cersat/products/gridded/mwf-quikscat/data/daily/1999/

and a typical file header and time value are

dimensions:
        time = 1 ;
        depth = 1 ;
        latitude = 320 ;
        longitude = 720 ;
variables:
        int time(time) ;
                time:long_name = "time" ;
                time:units = "hours since 1900-1-1 0:0:0" ;
        float depth(depth) ;
                depth:long_name = "depth" ;
                depth:units = "meters" ;
        float latitude(latitude) ;
                latitude:long_name = "latitude" ;
                latitude:units = "degrees_N" ;
        float longitude(longitude) ;
                longitude:long_name = "longitude" ;
                longitude:units = "degrees_E" ;
        short wind_stress(latitude, longitude) ;
                wind_stress:long_name = "wind stress magnitude" ;
                wind_stress:units = "Pa" ;
                wind_stress:_FillValue = 32767s ;
data:

time = 872796 ;

Applying ncecat to a single file or to a group of files gets rid of the time data, "872796" in the above example.

In the observational world, data files written like the above are very common.  Another variant of this is that
the time is written only in the file title, for example, 19991225.nc .

Todd

[Charlie Zender]

Todd,

For your problem, you should first convert 'time' from a fixed to a record dimension:

http://nco.sf.net/nco.html#dmn_rcd_mk

Then use ncrcat to concatenate the files.
Alternatively, you can use the 4.0.1 code for ncks which is now in CVS,
and use ncks -mk_rec_dmn time in.nc out.nc

CZ

[Todd Mitchell]

Hi CZ, and thank you for your help.

I tried both of your solutions and they did not work.

The first solution was to use the extant code for defining time as the record dimension, and then ncrcat to concatenate
the files.  Yes, time is defined as the record dimension by the first step, but the problem is that the data variable, for example zonal_wind_stress,
does not have time added as a dimension.  Application of ncrcat concatenates the time variable, but not zonal_wind_stress.
From an ncdump of the concatenated file: "double zonal_wind_stress(latitude, longitude) ;"   Another way to see this is the
size of one of the constituent files and of the concatenated file --- the small difference reflects the concatenation of only the time variable.
-rw-rw-r- 1 mitchell atgstaff 29502112 Apr  1 11:59 a.nc          %constituent file
-rw-rw-r- 1 mitchell atgstaff 29502292 Apr  1 09:35 out1.nc    %concatenated file

I went to http://nco.sourceforge.net/ and a search for "CVS" yielded "Releases receive unique CVS tags and their tarballs are stored here on SourceForge and here on my server. "  I went to the first link, http://nco.sourceforge.net/src/ ,
the computer staff installed nco_4.0.1.orig.tar.gz, I checked that it was ncks 4.0.1 and it didn't recognize -mk_rec_dmn

95 jisaoraid% ncks -version
NCO netCDF Operators version "4.0.1" last modified 2010/01/05 built Apr  1 2010 on jisaoraid by warren
ncks version 20100401
96 jisaoraid% ncks -h -O -mk_rec_dmn time a.nc b.nc
ncks: unrecognized option `-mk_rec_dmn'

Todd

[Charlie Zender]

1. then use ncecat to create a record dimension for the data variables, and rename "time" to "time_orig" and "record" to "time" then ncrcat will work.

or

2. the 4.0.1 code is not released yet. you found a pre-release version. you must check-out the code using CVS as
described on the homepage, build it, and then ncks will have -mk_rec_dmn

cz

[Todd Mitchell]

I am running NCO 4.0.2 last modified 2010/04/18

I tried suggestion 1. of reply #6 and the result is that the variable of interest gets concatenated, but the variable "time" does not,
and it appears as "time(time_orig)" in the ncdump output.  I have put an edited version of how the file evolves with processing at
ftp://ftp.atmos.washington.edu/pub/jisao/mitchell/nco</url]

Thank you for your help! 

[Charlie Zender]

Todd,

This is because ncecat does not concatenate coordinate variables.
You couldn't have known this because it was not mentioned on the ncecat man page.
I have just fixed this.
BTW, ncbo, ncea, and ncpdq (packing) do not operate on coordinate variables either.
This is a "feature" 99% of the time.
To work around it, if you want, just rename the variable so that it is not a coordinate before
operating with ncecat, then rename it back after ncecat.
Or, in your situation, you could ncwa -a time first, to remove the time dimension, then concatenate.
cz

[Dave Allured]

I was writing this when cz replied.  It seems to give the desired output for a case like this one, where each original file contains only one time step each.

ncwa -h -a time a1.nc b1.nc    # remove degenerate time dimension
ncwa -h -a time a2.nc b2.nc    # same for each input file
ncwa -h -a time a3.nc b3.nc    # etc.  (better to use some kind of shell loop)

ncecat b1.nc b2.nc b3.nc c.nc    # concatenate all files
ncrename -h -d record,time c.nc    # rename new "record" dimension to "time"

Step 4 has the desirable side effect of creating a record dimension, i.e. "unlimited".
Step 5 has the desirable side effect of converting the time variable from an ordinary 1-D data variable into the desired form of a 1-D coordinate variable.  I.e. the dimension name is identical to the var name.

BTW this should work for both older and newer versions of NCO.  HTH.

-Dave