I have been thinking about how to implement a unary square root
function in a quick and dirty way so we can get real standard
deviations and RMS computations done (without waiting for
ncap, which solves the problem more elegantly). I think the best
way to do this is to add the -y sqrt option to ncea or ncrcat.
Adding it to ncea has the advantage of working on record and non-
record variables. Then, e.g., ncea -y sqrt in.nc out.nc would
replace a file by its square root, and ncea -y sqrt in1.nc in2.nc out.nc
would write the square root of the average (which is useless,
but not harmful). The appropriate combination of
ncra,ncdiff, and ncea would then produce real standard deviations
and square roots. Any comments on this approach?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The previous post was a little of the mark.
It turns out that, without creating a new, speciality operator,
the easiest way to implement sdn is to add an rms option
which behaves like avgsumsqr and then follows up with
an sqrt operation. This seems to work. see new test case
in nco_tst.sh. So multifile timeseries sdn takes 5 steps!
but simple rms, which is useful in its own right, takes only
one step (the final step of sdn). Smoketest this everyone.
And see if it works with ensembles (ncea) as it should but
with caveat below.
Only know problem is that sdn normalizes by N not N-1.
This does not bother me as much as it should because I would
use this on long timeseries. But doing sdn on ensembles of files
where there are not many realizations would make this a real
problem. So what's needed is a patch to var_normalize() or
a new var_normalize_sdn() which subtracts one first and does
appropriate error checking. Note that this is where definitions
of rms and sdn would diverge--true rms should not normalize
by N-1, so perhaps new option rmssdn should be added to
indicate normalization by N-1? opinions?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm still unclear how you get from rms to sdn in a single step - Still if rms is simply sqrt(avgsumsqr) then it is trivial to implement. Normalization patch shouldn't be a problem
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, exactly, rms is implemented as sqrt(avgsumsqr).
Have a look, it's all committed. There were no changes
to ncwa so it shouldn't trip up what you are doing now.
If you want to write the normalization patch, please do.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I agree with you. The best place for the sqrt function is in ncea (ncra) along with the other functions. It will then be a 4/5 stage process to get the sdn. ( Using ncea -y avgsumsqr, and ncea -y avgsqr )
It isn't a very neat solution, but it should work.
Had a quick look at ncap. Looks very interesting, the lex and bison stuff !
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have been thinking about how to implement a unary square root
function in a quick and dirty way so we can get real standard
deviations and RMS computations done (without waiting for
ncap, which solves the problem more elegantly). I think the best
way to do this is to add the -y sqrt option to ncea or ncrcat.
Adding it to ncea has the advantage of working on record and non-
record variables. Then, e.g., ncea -y sqrt in.nc out.nc would
replace a file by its square root, and ncea -y sqrt in1.nc in2.nc out.nc
would write the square root of the average (which is useless,
but not harmful). The appropriate combination of
ncra,ncdiff, and ncea would then produce real standard deviations
and square roots. Any comments on this approach?
The previous post was a little of the mark.
It turns out that, without creating a new, speciality operator,
the easiest way to implement sdn is to add an rms option
which behaves like avgsumsqr and then follows up with
an sqrt operation. This seems to work. see new test case
in nco_tst.sh. So multifile timeseries sdn takes 5 steps!
ncrcat -O -v rec_var_flt_mss_val_dbl in.nc in.nc foo1.nc 2>> foo.tst
ncra -O -y avg -v rec_var_flt_mss_val_dbl in.nc in.nc foo2.nc 2>> foo.tst
ncwa -O -a time foo2.nc foo2.nc 2>> foo.tst
ncdiff -O -v rec_var_flt_mss_val_dbl foo1.nc foo2.nc foo2.nc 2>> foo.tst
ncra -O -y rms -v rec_var_flt_mss_val_dbl foo2.nc foo2.nc 2>> foo.tst
but simple rms, which is useful in its own right, takes only
one step (the final step of sdn). Smoketest this everyone.
And see if it works with ensembles (ncea) as it should but
with caveat below.
Only know problem is that sdn normalizes by N not N-1.
This does not bother me as much as it should because I would
use this on long timeseries. But doing sdn on ensembles of files
where there are not many realizations would make this a real
problem. So what's needed is a patch to var_normalize() or
a new var_normalize_sdn() which subtracts one first and does
appropriate error checking. Note that this is where definitions
of rms and sdn would diverge--true rms should not normalize
by N-1, so perhaps new option rmssdn should be added to
indicate normalization by N-1? opinions?
I'm still unclear how you get from rms to sdn in a single step - Still if rms is simply sqrt(avgsumsqr) then it is trivial to implement. Normalization patch shouldn't be a problem
Yes, exactly, rms is implemented as sqrt(avgsumsqr).
Have a look, it's all committed. There were no changes
to ncwa so it shouldn't trip up what you are doing now.
If you want to write the normalization patch, please do.
I agree with you. The best place for the sqrt function is in ncea (ncra) along with the other functions. It will then be a 4/5 stage process to get the sdn. ( Using ncea -y avgsumsqr, and ncea -y avgsqr )
It isn't a very neat solution, but it should work.
Had a quick look at ncap. Looks very interesting, the lex and bison stuff !