Hi, I am trying to find annual averages for global sea surface temperature data spread over 365 files, one for each day of the year. Dimensions are:
time = 1;
lon = 36000;
lat = 16000;
Geo2D variables containing these dimensions:
short analysed_sst;
byte mask;
short analysis_error;
I'm trying this with just two files first before writing a shell script for all the files, averaging only the analysed_sst variable:
Hi Leon,
Drop the "-d time,0" option, and/or make sure that "time" is a record dimension. ncra will only average variables with a record dimension. And you probably won't need a shell script. Something like
ncra in*.nc out.nc
should suffice.
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie, and thanks for the prompt reply. I think I was having trouble with the concept of a record dimension, it doesn't seem to be defined in the data anyway. I was able to generate output using nces instead, which is probably what I wanted to do in the first place:
nces -v analysed_sst 2011010*.nc out.nc
My next problem is to do this for 365 files instead of just 2. My VM is around 300Gb, but each file decompresses to around 3Gb - not enough space. I wonder if there is some method in NCO to iterate across the bzipped files which are around 120Mb each, decompressing each one in turn to NC, reading it, adding its data to the running average and then deleting the NC before moving on to the next one?
Thanks again for the help!
Leon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Don't give up on the easy solution until you try the command for the full dataset and it doesn't work. NCO is parsimonious with memory so it may just work. Use -D 2 so you can keep track of the progress of the command.
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie, actually I'm not too worried about RAM usage - I meant my VM only has 300Gb of hard drive storage available, not enough to decompress all the data files at once. Is there a way to bunzip the files one at a time in NCO or with a shell script?
Cheers
Leon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You'll need to write a shell script to bzip/bunzip. NCO has a feature to obtain, process, then delete remotely-retrieved files. It may be useful: http://nco.sf.net/nco.html#rmt
good luck,
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi, I am trying to find annual averages for global sea surface temperature data spread over 365 files, one for each day of the year. Dimensions are:
time = 1;
lon = 36000;
lat = 16000;
Geo2D variables containing these dimensions:
short analysed_sst;
byte mask;
short analysis_error;
I'm trying this with just two files first before writing a shell script for all the files, averaging only the analysed_sst variable:
ncra.exe -d time,0 -v analysed_sst 20110101-JPL_OUROCEAN-L4UHfnd-GLOB-v01-fv01_0-G1SST.nc 20110102-JPL_OUROCEAN-L4UHfnd-GLOB-v01-fv01_0-G1SST.nc output.nc
ncra: ERROR no variables fit criteria for processing
How do I target the variable properly? Do I need to change the variable type first somehow?
Thanks for this great tool! Imagine, I was trying to do this in R before...
Leon
Hi Leon,
Drop the "-d time,0" option, and/or make sure that "time" is a record dimension. ncra will only average variables with a record dimension. And you probably won't need a shell script. Something like
should suffice.
cz
Hi Charlie, and thanks for the prompt reply. I think I was having trouble with the concept of a record dimension, it doesn't seem to be defined in the data anyway. I was able to generate output using nces instead, which is probably what I wanted to do in the first place:
My next problem is to do this for 365 files instead of just 2. My VM is around 300Gb, but each file decompresses to around 3Gb - not enough space. I wonder if there is some method in NCO to iterate across the bzipped files which are around 120Mb each, decompressing each one in turn to NC, reading it, adding its data to the running average and then deleting the NC before moving on to the next one?
Thanks again for the help!
Leon
Don't give up on the easy solution until you try the command for the full dataset and it doesn't work. NCO is parsimonious with memory so it may just work. Use -D 2 so you can keep track of the progress of the command.
cz
Hi Charlie, actually I'm not too worried about RAM usage - I meant my VM only has 300Gb of hard drive storage available, not enough to decompress all the data files at once. Is there a way to bunzip the files one at a time in NCO or with a shell script?
Cheers
Leon
You'll need to write a shell script to bzip/bunzip. NCO has a feature to obtain, process, then delete remotely-retrieved files. It may be useful: http://nco.sf.net/nco.html#rmt
good luck,
cz