I am new working with NetCDF and NCO and I am trying to format a time 2D variable for my dataset with observation number on X-axis and datetime on Y-axis with length 6 as: {yy,mm,dd,HH,MM,SS}. Represented as a table array would be:
2019|2019 5|51|112|120|00|4
I am able to achieve this format but only when using a predefined list of values {2019,5,1,0,4} inside the next .nco script:
When trying to define this list of values as a function of a variable "t" as date[date]=t; that will be later introduced in a loop in the command line, then the values are added but when trying to display the values by using Panoply software an error occur: "Failed creating the data handler: Axis date does not have unique values; found consecutive 2019.0 values. ." The command line would be: ncap2 -Oh -s "t={2019,10,19,1,2,3};" -S test.nco in.nc out.nc
I am not getting this error when adding the values directly as a list in the script
Does anyone has an idea why I am getting this error?
Thank you in advance,
Pablo
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Please resubmit this post after using the widgets in the edit toolbar to clearly mark code as code so characters are aligned in a monospace font that shows more exactly what you have typed.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am new working with NetCDF and NCO and I am trying to format a time 2D variable for my dataset with observation number on X-axis and datetime on Y-axis with length 6 as: {yy,mm,dd,HH,MM,SS}. Represented as a table array would be:
I am able to achieve this format but only when using a predefined list of values {2019,5,1,0,4} inside the next .nco script:
/***.nco ncap2 script***/defdim("nobs",2);defdim("date",6);nobs[nobs]={1,2};date[date]={2019,5,1,0,4};yymmddHHMMSS[date,nobs]=date[date];yymmddHHMMSS@values=date[date];yymmddHHMMSS@long_name="Time in yymmddHHMMSS format";yymmddHHMMSS@axis="T";yymmddHHMMSS@coordinate_defines="point";
When trying to define this list of values as a function of a variable "t" as date[date]=t; that will be later introduced in a loop in the command line, then the values are added but when trying to display the values by using Panoply software an error occur: "Failed creating the data handler: Axis date does not have unique values; found consecutive 2019.0 values. ." The command line would be: ncap2 -Oh -s "t={2019,10,19,1,2,3};" -S test.nco in.nc out.nc
I am not getting this error when adding the values directly as a list in the script
Does anyone has an idea why I am getting this error?
Thank you in advance,
Pablo Reyes
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It is not clear what you want to achieve,
although I suspect you want to create a date coordinate variable
that encapsulates SS, MM, HH, dd, mm, yy.
Note that coordinate variables are one-dimensional and monotonic.
If the date-time in your tabular data is not monotonic, you would need
to sort them in increasing or decreasing order of yy,mm,dd,HH,MM,SS
Also, it may help if you tell how many observations you have,
how many years, months, hours, minutes, seconds in
your tabular data.
INetCDF the dimensions are just scalars.
When you say
defdim("nobs",2)
defdim("date",6)
an array with dimensions (nobs,date) will be a 2x6 array.
When you say
nobs[nobs]={1,2};date[date]={2019,5,1,0,4};
you are creating the corresponding coordinate variables,
with specific values. You fully populated coordinate variable nobs
with two values (the nobs dimension size is 2) 1 and 2,
and you populated 5 of the 6 possible values of date with
2019,5,1,0, and 4.
My impression is that this is not what you want to achieve.
but you need to sort your table in increasing order of (SS,MM,HH,dd,mm,yy)
anyway.
Knowing date you can recover each of SS, MM, etc, using integer division
and the mod (module or remainder) function.
I am new working with NetCDF and NCO and I am trying to format a time 2D
variable for my dataset with observation number on X-axis and datetime on
Y-axis with length 6 as: {yy,mm,dd,HH,MM,SS}. Represented as a table array
would be:
I am able to achieve this format but only when using a predefined list of
values {2019,5,1,0,4} inside the next .nco script:
/.nco ncap2 script/defdim("nobs",2);defdim("date",6);nobs[nobs]={1,2};date[date]={2019,5,1,0,4};yymmddHHMMSS[date,nobs]=date[date];yymmddHHMMSS@values=date[date];yymmddHHMMSS@long_name="Time in yymmddHHMMSS format";yymmddHHMMSS@axis="T";yymmddHHMMSS@coordinate_defines="point";
When trying to define this list of values as a function of a variable "t"
as date[date]=t; that will be later introduced in a loop in the command
line, then the values are added but when trying to display the values by
using Panoply software an error occur: "Failed creating the data handler:
Axis date does not have unique values; found consecutive 2019.0 values. ."
The command line would be: ncap2 -Oh -s "t={2019,10,19,1,2,3};" -S test.nco
in.nc out.nc
I am not getting this error when adding the values directly as a list in
the script
Does anyone has an idea why I am getting this error?
Thank you very much for your detailed answer and your suggestions. I am also taking a look to that links. My intention was just to create a variable to later relate the sample number with the date in that format. Maybe It doesn't have to be a standard coordinate variable, although I also understand that maybe that was not the standard way to proceed with the date.
On the other hand, it's true that I missed one digit in my code example above. I defined a 6 dimension date for dd-mm-yy-HH-MM-SS and only provided 5 numbers, that was just a mistake, sorry. My sample size is around 4 years of daily data, that is around 1460 observations. As I said, I can achieve this as a test when populating directly the variable from the script, but I would like to do this in an automatic way from the terminal. When adding this values from the terminal as:
netCDF variables are not array-valued as you seem to want them to be
when you try to define the variable yymmddHHMMSS.
The variables are scalar-valued.
Please check the links on netCDF file structure, components, requirements
that I sent you in the previous email.
If your observation number/index is the primary key to your data,
you may want to have separate variables for SS,MM,HH,dd,mm,yy,
all indexed by the observation number,
along with the observation data variable,
and along with an unlimited coordinate variable nobs[nobs].
The closest thing to an yymmddHHMMSS variable would be, as I suggested, to
encapsulate the
date and time information with something like this (integer variable):
To avoid overflowing the maximum integer (2,147,483,647), you may want to
subtract the reference year (2019?) from yy above.
This way SS=mod(date,60), MM= mod(date-SS,60), with similar formulas for
HH, etc.
Thank you very much for your detailed answer and your suggestions. I am
also taking a look to that links. My intention was just to create a
variable to later relate the sample number with the date in that format.
Maybe It doesn't have to be a standard coordinate variable, although I also
understand that maybe that was not the standard way to proceed with the
date.
On the other hand, it's true that I missed one digit in my code example
above. I defined a 6 dimension date for dd-mm-yy-HH-MM-SS and only provided
5 numbers, that was just a mistake, sorry. My sample size is around 4 years
of daily data, that is around 1460 observations. As I said, I can achieve
this as a test when populating directly the variable from the script, but I
would like to do this in an automatic way from the terminal. When adding
this values from the terminal as:
Hi there,
I am new working with NetCDF and NCO and I am trying to format a time 2D variable for my dataset with observation number on X-axis and datetime on Y-axis with length 6 as: {yy,mm,dd,HH,MM,SS}. Represented as a table array would be:
I am able to achieve this format but only when using a predefined list of values {2019,5,1,0,4} inside the next .nco script:
/.nco ncap2 script/
defdim("nobs",2);
defdim("date",6);
nobs[nobs]={1,2};
date[date]={2019,5,1,0,4};
/date[date]=t;/
yymmddHHMMSS[date,nobs]=date[date];
yymmddHHMMSS@values=date[date];
yymmddHHMMSS@long_name="Time in yymmddHHMMSS format";
yymmddHHMMSS@axis="T";
yymmddHHMMSS@coordinate_defines="point";
When trying to define this list of values as a function of a variable "t" as date[date]=t; that will be later introduced in a loop in the command line, then the values are added but when trying to display the values by using Panoply software an error occur: "Failed creating the data handler: Axis date does not have unique values; found consecutive 2019.0 values. ." The command line would be: ncap2 -Oh -s "t={2019,10,19,1,2,3};" -S test.nco in.nc out.nc
I am not getting this error when adding the values directly as a list in the script
Does anyone has an idea why I am getting this error?
Thank you in advance,
Pablo
Please resubmit this post after using the widgets in the edit toolbar to clearly mark code as code so characters are aligned in a monospace font that shows more exactly what you have typed.
Hi there,
I am new working with NetCDF and NCO and I am trying to format a time 2D variable for my dataset with observation number on X-axis and datetime on Y-axis with length 6 as: {yy,mm,dd,HH,MM,SS}. Represented as a table array would be:
I am able to achieve this format but only when using a predefined list of values {2019,5,1,0,4} inside the next .nco script:
When trying to define this list of values as a function of a variable "t" as date[date]=t; that will be later introduced in a loop in the command line, then the values are added but when trying to display the values by using Panoply software an error occur: "Failed creating the data handler: Axis date does not have unique values; found consecutive 2019.0 values. ." The command line would be: ncap2 -Oh -s "t={2019,10,19,1,2,3};" -S test.nco in.nc out.nc
I am not getting this error when adding the values directly as a list in the script
Does anyone has an idea why I am getting this error?
Thank you in advance,
Pablo Reyes
Hi Pablo
It is not clear what you want to achieve,
although I suspect you want to create a date coordinate variable
that encapsulates SS, MM, HH, dd, mm, yy.
Note that coordinate variables are one-dimensional and monotonic.
If the date-time in your tabular data is not monotonic, you would need
to sort them in increasing or decreasing order of yy,mm,dd,HH,MM,SS
Also, it may help if you tell how many observations you have,
how many years, months, hours, minutes, seconds in
your tabular data.
INetCDF the dimensions are just scalars.
When you say
defdim("nobs",2)
defdim("date",6)
an array with dimensions (nobs,date) will be a 2x6 array.
When you say
nobs[nobs]={1,2};date[date]={2019,5,1,0,4};
you are creating the corresponding coordinate variables,
with specific values. You fully populated coordinate variable nobs
with two values (the nobs dimension size is 2) 1 and 2,
and you populated 5 of the 6 possible values of date with
2019,5,1,0, and 4.
My impression is that this is not what you want to achieve.
Note that the coordinate variable date above is not monotonic, as it
should have been.
https://www.unidata.ucar.edu/software/netcdf/workshops/2011/datamodels/NcCVars.html
It decreases at first, then increases again (from 0 to 4).
You may be able to do what you want with NCO, but since your goal is not clear,
it is hard to tell.
Have you tried to do this with NCL?
NCL has useful date functions that may help.
https://www.ncl.ucar.edu/
I'd suggest also that first you get a bit familiar with the basic
structure, components, and requirements
of a netCDF file (dimensions, coordinate variables, variables,
attributes, data types, etc).
https://www.unidata.ucar.edu/software/netcdf/workshops/2011/datamodels/index.html
That may help you translate what you want to do onto the netCDF data
structure and components.
You could also try to use an unlimited date coordinate, and
single integer date coordinate variable like this:
date = SS + 60 * MM + 60 * 60 * HH + 24 * 60 * 60 * dd + 31 * 24 * 60 * 60
* mm + 12 * 31 * 24 * 60 * 60 * yy
but you need to sort your table in increasing order of (SS,MM,HH,dd,mm,yy)
anyway.
Knowing date you can recover each of SS, MM, etc, using integer division
and the mod (module or remainder) function.
I hope this helps,
Gus Correa
On Thu, Mar 11, 2021 at 5:57 PM Pablo psreyes@users.sourceforge.net wrote:
Hi Gus,
Thank you very much for your detailed answer and your suggestions. I am also taking a look to that links. My intention was just to create a variable to later relate the sample number with the date in that format. Maybe It doesn't have to be a standard coordinate variable, although I also understand that maybe that was not the standard way to proceed with the date.
On the other hand, it's true that I missed one digit in my code example above. I defined a 6 dimension date for dd-mm-yy-HH-MM-SS and only provided 5 numbers, that was just a mistake, sorry. My sample size is around 4 years of daily data, that is around 1460 observations. As I said, I can achieve this as a test when populating directly the variable from the script, but I would like to do this in an automatic way from the terminal. When adding this values from the terminal as:
ncap2 -Oh -s "t={2019,19,10,5,1,0}" -S time.nco in.nc test.ncit does not read the numbers in the list, but values are stored like:
My script reads dates from the files in a loop, as for example: VARIABLE_01012019.nc and tries to add this date to the 2D date variable.
Anyway, I appreciate your suggestions and helps me a lot to clarify the ideas.
Thanks,
Pablo Reyes
Pablo
netCDF variables are not array-valued as you seem to want them to be
when you try to define the variable yymmddHHMMSS.
The variables are scalar-valued.
Please check the links on netCDF file structure, components, requirements
that I sent you in the previous email.
If your observation number/index is the primary key to your data,
you may want to have separate variables for SS,MM,HH,dd,mm,yy,
all indexed by the observation number,
along with the observation data variable,
and along with an unlimited coordinate variable nobs[nobs].
ii=1
nobs[ii]=ii
yy[ii]=2019
mm[ii]=5
dd[ii]=1
HH[ii]=0
MM[ii]=0
SS[ii]=0
data[ii]=first_observation_value
ii=2
nobs[ii]=ii
yy[ii]=2019
mm[ii]=5
dd[ii]=1
HH[ii]=0
MM[ii]=30
SS[ii]=0
data[ii]=second_observation_value
etc
The closest thing to an yymmddHHMMSS variable would be, as I suggested, to
encapsulate the
date and time information with something like this (integer variable):
date[ii] = SS + 60 * MM + 60 * 60 * HH + 24 * 60 * 60 * dd + 31 * 24 * 60 *
60 * mm + 12 * 31 * 24 * 60 * 60 * yy
To avoid overflowing the maximum integer (2,147,483,647), you may want to
subtract the reference year (2019?) from yy above.
This way SS=mod(date,60), MM= mod(date-SS,60), with similar formulas for
HH, etc.
I hope this helps.
Gus
On Sat, Mar 13, 2021 at 6:15 AM Pablo psreyes@users.sourceforge.net wrote: