ncwa returns only temporary file, no error message

[Sjur Kolberg]

Hi,

I am trying to calculate an annual average from five years of hourly precipitation data residing in the same file. The time dimension is not a record dimension, so I use ncwa:

C:\Prosjekter\EnkiDemo_TEK\EnkiSim>\apps\nco\ncwa -v SeNorgeCorrPrec -a time,SeNorgeCorrPrecparset,SeNorgeCorrPrecldtime,SeNorgeCorrPrecmember -d time,8761,52584 Fullinput.nc avgP1015.nc

The command returns happily after a while, as if all is OK.
But on disk I find only a temporary file named avgP1015.nc.pid23804.ncwa.tmp.

I also tried the same command, but without mentioning the three exta dimensions with the long names. These are all length=1, non-record dims. No difference.
I also tried to use time coordinates rather than indices, causing error code = -40 from ncwa.

Not much nco experience, is anyone seeing what I am doing wrong?

I can rename the temporary file to avgP1015.nc and open it in Panoply, which recognises the file structure (dims and variables, all attributes seem OK). My SeNorgeCorrPrec variable is recognised as 'Geo2D'. Its metadata are displayed as expected:
float SeNorgeCorrPrec(y=128, x=114);
:_FillValue = -999.99f; // float
:y = "y";
:x = "x";
:coordinates = "lon lat";
:units = "";
:grid_mapping = "utm_32n";
:time = "time";
:parset = "SeNorgeCorrPrecparset";
:ldtime = "SeNorgeCorrPrecldtime";
:member = "SeNorgeCorrPrecmember";
:cell_methods = "time, SeNorgeCorrPrecparset, SeNorgeCorrPrecldtime, SeNorgeCorrPrecmember: mean";
:_ChunkSizes = 128U, 114U; // uint

When I try to plot it, however, Panoply complains: There was an error preparing the plot. Failed creating the data handler: Axis includes NaN value(s).
And indeed, the x and y coordinate variables cannot be plotted either; same error message.

Also the SAGA GIS software is able to open the file, find the variable, and even create a plot.
However, the value in every grid cell is 9969209968386869046778552952102584320.00.
The spatial coordinates, cell size or georeferencing is not recognised; I just get a rectangle with the right dimensions. SAGA is fine with all this, though, and acts as if the huge cell values are perfectly normal.

So there seems to be two issues, the spatial coordinates and the cell values.

Please don't bother with proper scaling or sum vs average; if I only get the results I can do the scaling elsewhere. The georeferencing can also be restored afterwards. I know from other use that the spatial information is present in the origninal input file. The variable in question does have missing values, but not within the requested period.

I'm on Windows 10:
C:\Prosjekter\EnkiDemo_TEK\EnkiSim>\apps\nco\ncwa --version
NCO netCDF Operators version 4.7.7-alpha01 built by pvn on DESKTOP-M3AK925 at Aug 28 2018 12:47:05
ncwa version 4.7.7-alpha01

I am grateful for any help.

Sjur K :-)

Verbose output from running with -D option below:

C:\Prosjekter\EnkiDemo_TEK\EnkiSim>\apps\nco\ncwa -D 5 -v SeNorgeCorrPrec -a time,SeNorgeCorrPrecparset,SeNorgeCorrPrecldtime,SeNorgeCorrPrecmember -d time,8761,52584 Fullinput.nc avgP1015.nc
ncwa: INFO nco_cnk_csh_ini() reports chunk cache size = cnk_csh_byt = 4194304 B, # of slots in raw data chunk cache has table = nelemsp = 1009, pre-emption favor fraction = pmp_fvr_frc = 0.75
ncwa: INFO Build compiler lacked (or user turned-off) OpenMP support. Code will execute with single thread in Uni-Processor (UP) mode.
ncwa: INFO nco_fl_open() reports nc__open() will request file buffer of default size
ncwa: INFO nco_fl_open() reports nc__open() opened file with buffer size = 0 bytes
ncwa: INFO nco_fl_open() reports extended filetype of Fullinput.nc is NC_FORMATX_NC_HDF5, mode = 10000 (oct) = 4096 (dec) = 1000 (hex)
ncwa: INFO nco_fl_open() will not print any more INFO messages if this file is opened again. (Many NCO operators open the same file multiple times when OpenMP is enabled, nco_fl_open() prints INFO messages only the first time because successive messages are usually redundant).
ncwa: CONVENTION File "Conventions" attribute is "CF-1.6"
ncwa: INFO NCO attempts to abide by many official and unofficial metadata conventions including ARM, CCM, CCSM, and CF. To adhere to these conventions, NCO implements variable-specific exceptions in certain operators, e.g., ncbo will not subtract variables named "date" or "gw", and many operators will always leave coordinate variables unchanged. The full list of exceptions is in the manual http://nco.sf.net/nco.html#CF
ncwa: INFO dimensions to average: <senorgecorrprecparset><senorgecorrprecldtime><senorgecorrprecmember>
ncwa: CONVENTION File "Conventions" attribute is "CF-1.6"
ncwa: INFO NCO attempts to abide by many official and unofficial metadata conventions including ARM, CCM, CCSM, and CF. To adhere to these conventions, NCO implements variable-specific exceptions in certain operators, e.g., ncbo will not subtract variables named "date" or "gw", and many operators will always leave coordinate variables unchanged. The full list of exceptions is in the manual http://nco.sf.net/nco.html#CF
var[0]->nm = SeNorgeCorrPrec, ->id=[21]
var[1]->nm = SeNorgeCorrPrecldtime, ->id=[22]
var[2]->nm = SeNorgeCorrPrecmember, ->id=[23]
var[3]->nm = SeNorgeCorrPrecparset, ->id=[24]
var[4]->nm = lat, ->id=[37]
var[5]->nm = lon, ->id=[39]
var[6]->nm = time, ->id=[42]
var[7]->nm = utm_32n, ->id=[43]
var[8]->nm = x, ->id=[44]
var[9]->nm = y, ->id=[45]
var_fix[0]->nm = lat, ->id=[37]
var_fix[1]->nm = lon, ->id=[39]
var_fix[2]->nm = utm_32n, ->id=[43]
var_fix[3]->nm = x, ->id=[44]
var_fix[4]->nm = y, ->id=[45]
var_prc[0]->nm = SeNorgeCorrPrec, ->id=[21]
var_prc[1]->nm = SeNorgeCorrPrecldtime, ->id=[22]
var_prc[2]->nm = SeNorgeCorrPrecmember, ->id=[23]
var_prc[3]->nm = SeNorgeCorrPrecparset, ->id=[24]
var_prc[4]->nm = time, ->id=[42]
ncwa: INFO nco_fl_blocksize() reports preferred output filesystem I/O block size: 0 bytes
ncwa: INFO User requests (or compression or record dimension requires) chunking or unchunking
cnk_plc, cnk_map: xst, xst
cnk_sz_scl, cnk_sz_byt, cnk_min_byt: 0, 4194304, 8192
ncwa: INFO nco_cnk_sz_set_trv() SeNorgeCorrPrec must be chunked (record, compressed, or check-summed variable)
ncwa: INFO nco_cnk_sz_set_trv() re-chunking SeNorgeCorrPrec
idx dmn_nm dmn_sz cnk_sz:
0 /y 128 128
1 /x 114 114
ncwa: INFO nco_aed_prc() reports attempt to create, modify, or overwrite SeNorgeCorrPrec attribute _FillValue in netCDF4 file violates netCDF4 capabilities (and would result in NC_ELATEFILL error) so will invoke NCO hocus-pocus rename trick...
ncwa: INFO nco_cnk_sz_set_trv() not unchunking lat because it is not chunked
ncwa: INFO nco_cnk_sz_set_trv() not unchunking lon because it is not chunked
ncwa: INFO nco_cnk_sz_set_trv() not unchunking utm_32n because it is not chunked
ncwa: INFO nco_cnk_sz_set_trv() not unchunking x because it is not chunked
ncwa: INFO nco_cnk_sz_set_trv() not unchunking y because it is not chunked
ncwa: INFO Input file 0 is Fullinput.nc
ncwa: TIMER Metadata setup and file layout before main loop took 0.21 s
ncwa: INFO main loop processing var_prc[0] = "SeNorgeCorrPrec"
ncwa: Promoting variable SeNorgeCorrPrec from type NC_FLOAT to type NC_DOUBLE
ncwa: Promoting variable SeNorgeCorrPrec from type NC_FLOAT to type NC_DOUBLE</senorgecorrprecmember></senorgecorrprecldtime></senorgecorrprecparset>

C:\Prosjekter\EnkiDemo_TEK\EnkiSim>

[Charlie Zender]

I do not have Windows so I am flying blind. The command looks OK. If a dimension has size=1 then no need to include it in the averaging list, though it should not hurt. That said, your command looks good to me. I suggest starting small, and averaging smaller variables or in smaller pieces to see where things go wrong. Also it might be helpful to try this on Linux to see if it is an OS issue.

[Sjur Kolberg]

Charlie, thanks for your advice. I have only Windows, and do not feel competent to start debugging. I worked around the problem by first making time a record variable and then using ncra instead. Geoinformation is still not coming through, but fixing that afterwards is a breeze compared to summing five years of hourly data outside nco :-)