ncap2 over every parameter

[dominique briand]

Hello, I want to higlight for every parameter of my netcdf file how many points are not egal to 0. I'm wondering if the command ncap2 allows me for doing that. I already do it manually for 1 parameter of my netcdf file, but I want to run it automatically over every parameter...

ncap2 -s 'where(SSH!=0)SSH=1;print(SSH.total());' SM_OPER_MIR_diff.DBL test.DBL

Do I have to write a script for thar?

[henry Butowsky]

Hi Dominque,
Its a little tricky but you can loop over the variables in the input file using what we call variable pointers.
Please look at the examples in nco/data -
-rw-rw-r-- 1 hmb hmb 2105 Apr 10 11:48 vpointer-ex3.nco
-rw-rw-r-- 1 hmb hmb 4096 Nov 30 15:45 vpointer-ex0.nco
-rw-rw-r-- 1 hmb hmb 920 Nov 30 15:45 vpointer-ex1.nco
-rw-rw-r-- 1 hmb hmb 1795 Nov 30 15:45 vpointer-ex2.nco
-rw-rw-r-- 1 hmb hmb 545 Nov 30 15:45 vpointer-tst.nco

The script looks for var types that are NC_DOUBLE and that have at least one dimension:
But it can be easily modified

to run the script use the command

ncap2 -v -O -S vpointer.nco SM_OPER_MIR_diff.DBL test.DBL

below is the script vpointer-ex.nco

`
// Produce a list of all vars in input
// each element of att list is of type NC_STRING
// nb get_var_in() can take a single argument that can be a var name or regular expression
// lists are of type NC_STRING and can are in essence ragged arrays
@all=get_vars_in();
sz=@all.size();
idx=0;

for(idx=0;idx<sz;idx++)
{
// remember @var_nm is of type NC_STRING
@var_nm=@all(idx);

if( @var_nm.type()==NC_DOUBLE && @var_nm.ndims()>1 ){
where( @var_nm !=0 )
@var_nm=1;

 print(*@var_nm.total());

}
}

`
...Henry

[henry Butowsky]

The format is a little messed up
here sthe script again.

[henry Butowsky]

Here it is

[dominique briand]

ok, thanks! It helps, but sometimes I have negative numbers :/. I don't understand...
How does this script lead with Na values?

@all=get_vars_in();
sz=@all.size();
idx=0;
print(sz);

for(idx=0;idx<sz;idx++)
{
// remember @var_nm is of type NC_STRING
@var_nm=@all(idx);
if(exists(@var_nm))
{
where(@var_nm!=0)@var_nm=1;
print(@var_nm.total());

}
}

[dominique briand]

Actually it seems my Na values ( that appear as _ when i dump) induce this problem,when I replace my _ by 0 (ncatted -a FillValue,NRfiX,d,d,1.0) it seems to work well. I'm gonna try to add this in the script

[dominique briand]

actually my results are totaly different...:/, I m missing something!

[henry Butowsky]

HI Dominque,
There are a couple of issues at play.
Are all the vars you want to process all the same type ?
do they all have the same FillValue ?
The '' in an ncdump indicates a fill value ?
If you file is not too large can you pls post it to the forum and ill take a look
....Henry

[dominique briand]

I just paste here the header of one of my files, the vars are not the same type, the fillValue are different...

I read that the elements with _fillValue are not taken into account, so my *@var_nm.total() should avoid them...

dimensions:
inc = 15 ;
lat = 584 ;
lon = 1388 ;
variables:
double Azimuth_Angle(inc, lat, lon) ;
Azimuth_Angle:_FillValue = -32768. ;
Azimuth_Angle:long_name = "Angle class averaged Azimuth angle value corresponding to the measured BT value over current Earth fixed grid point. Measured as angle in pixel local tangent plane from projected pixel to S/C direction with respect to the local North (0 if local North)" ;
Azimuth_Angle:units = "degrees" ;
double BT_3(inc, lat, lon) ;
BT_3:_FillValue = -32768. ;
BT_3:long_name = "Angle class averaged Brightness temperature 3rd Stokes parameter over current Earth fixed grid point, obtained by polarisation rotation from L1c data" ;
BT_3:units = "K" ;
double BT_4(inc, lat, lon) ;
BT_4:_FillValue = -32768. ;
BT_4:long_name = "Angle class averaged Brightness temperature 4th Stokes parameter over current Earth fixed grid point, obtained by polarisation rotation from L1c data" ;
BT_4:units = "K" ;
double BT_H(inc, lat, lon) ;
BT_H:_FillValue = -32768. ;
BT_H:long_name = "Angle class averaged Brightness temperature in H-pol over current Earth fixed grid point, obtained by polarisation rotation from L1c data" ;
BT_H:units = "K" ;
double BT_V(inc, lat, lon) ;
BT_V:_FillValue = -32768. ;
BT_V:long_name = "Angle class averaged Brightness temperature in V-pol over current Earth fixed grid point, obtained by polarisation rotation from L1c data" ;
BT_V:units = "K" ;
int Days(inc, lat, lon) ;
Days:_FillValue = -2147483647 ;
Days:long_name = "UTC Time at which the averaged BT was taken, in EE CFI transport time format. Days" ;
Days:units = "d" ;
double Eta(inc, lat, lon) ;
Eta:_FillValue = -32768. ;
Eta:long_name = "Angle class averaged Direction cosine : Eta" ;
Eta:units = "NA" ;
double Footprint_Axis1(inc, lat, lon) ;
Footprint_Axis1:_FillValue = -32768. ;
Footprint_Axis1:long_name = "Angle class averaged Elliptical footprint major semi-axis value" ;
Footprint_Axis1:units = "km" ;
double Footprint_Axis2(inc, lat, lon) ;
Footprint_Axis2:_FillValue = -32768. ;
Footprint_Axis2:long_name = "Angle class averaged Elliptical footprint minor semi-axis value" ;
Footprint_Axis2:units = "km" ;
byte Grid_Point_Mask(lat, lon) ;
Grid_Point_Mask:_FillValue = 0b ;
Grid_Point_Mask:long_name = "Grid_Point_Mask" ;
Grid_Point_Mask:units = "NA" ;
double Incidence_Angle(inc, lat, lon) ;
Incidence_Angle:_FillValue = -32768. ;
Incidence_Angle:long_name = "Angle class averaged Incidence angle value corresponding to the measured BT value over current Earth fixed grid point. Measured as angle from pixel to S/C with respect to the pixel local normal (0 if vertical)" ;
Incidence_Angle:units = "degrees" ;
short Nviews(inc, lat, lon) ;
Nviews:_FillValue = -32768s ;
Nviews:long_name = "Number of views used to compute Angle class averages" ;
Nviews:units = "NA" ;
double Pixel_BT_Standard_Deviation_3(inc, lat, lon) ;
Pixel_BT_Standard_Deviation_3:_FillValue = -32768. ;
Pixel_BT_Standard_Deviation_3:long_name = "Angle class BT standard deviation in the Brightness Temperature presented in the previous field, extracted in the direction of the pixel " ;
Pixel_BT_Standard_Deviation_3:units = "K" ;
double Pixel_BT_Standard_Deviation_4(inc, lat, lon) ;
Pixel_BT_Standard_Deviation_4:_FillValue = -32768. ;
Pixel_BT_Standard_Deviation_4:long_name = "Angle class BT standard deviation in the Brightness Temperature presented in the previous field, extracted in the direction of the pixel " ;
Pixel_BT_Standard_Deviation_4:units = "K" ;
double Pixel_BT_Standard_Deviation_H(inc, lat, lon) ;
Pixel_BT_Standard_Deviation_H:_FillValue = -32768. ;
Pixel_BT_Standard_Deviation_H:long_name = "Angle class BT standard deviation in the Brightness Temperature presented in the previous field, extracted in the direction of the pixel " ;
Pixel_BT_Standard_Deviation_H:units = "K" ;
double Pixel_BT_Standard_Deviation_V(inc, lat, lon) ;
Pixel_BT_Standard_Deviation_V:_FillValue = -32768. ;
Pixel_BT_Standard_Deviation_V:long_name = "Angle class BT standard deviation in the Brightness Temperature presented in the previous field, extracted in the direction of the pixel " ;
Pixel_BT_Standard_Deviation_V:units = "K" ;
double Pixel_Radiometric_Accuracy_3(inc, lat, lon) ;
Pixel_Radiometric_Accuracy_3:_FillValue = -32768. ;
Pixel_Radiometric_Accuracy_3:long_name = "Angle class averaged Error accuracy measurement in the brightness Temperature presented in the previous field, extracted in the direction of the pixel" ;
Pixel_Radiometric_Accuracy_3:units = "K" ;
double Pixel_Radiometric_Accuracy_4(inc, lat, lon) ;
Pixel_Radiometric_Accuracy_4:_FillValue = -32768. ;
Pixel_Radiometric_Accuracy_4:long_name = "Angle class averaged Error accuracy measurement in the brightness Temperature presented in the previous field, extracted in the direction of the pixel" ;
Pixel_Radiometric_Accuracy_4:units = "K" ;
double Pixel_Radiometric_Accuracy_H(inc, lat, lon) ;
Pixel_Radiometric_Accuracy_H:_FillValue = -32768. ;
Pixel_Radiometric_Accuracy_H:long_name = "Angle class averaged Error accuracy measurement in the brightness Temperature presented in the previous field, extracted in the direction of the pixel " ;
Pixel_Radiometric_Accuracy_H:units = "K" ;
double Pixel_Radiometric_Accuracy_V(inc, lat, lon) ;
Pixel_Radiometric_Accuracy_V:_FillValue = -32768. ;
Pixel_Radiometric_Accuracy_V:long_name = "Angle class averaged Error accuracy measurement in the brightness Temperature presented in the previous field, extracted in the direction of the pixel" ;
Pixel_Radiometric_Accuracy_V:units = "K" ;
int UTC_Microseconds(inc, lat, lon) ;
UTC_Microseconds:_FillValue = -2147483647 ;
UTC_Microseconds:long_name = "UTC Time at which the averaged BT was taken, in EE CFI transport time format. Microseconds" ;
UTC_Microseconds:units = "10-6s" ;
int UTC_Seconds(inc, lat, lon) ;
UTC_Seconds:_FillValue = -2147483647 ;
UTC_Seconds:long_name = "UTC Time at which the averaged BT was taken, in EE CFI transport time format. Seconds" ;
UTC_Seconds:units = "s" ;
float X_Swath(lat, lon) ;
X_Swath:_FillValue = 9.96921e+36f ;
X_Swath:long_name = "Minimum distance of grid point to the sub satellite point track" ;
X_Swath:units = "m" ;
double Xi(inc, lat, lon) ;
Xi:_FillValue = -32768. ;
Xi:long_name = "Angle class averaged Direction cosine: Xi" ;
Xi:units = "NA" ;
float dinc(inc) ;
dinc:_FillValue = 9.96921e+36f ;
dinc:long_name = "Width of each class of angles" ;
dinc:units = "degrees" ;
float inc(inc) ;
inc:_FillValue = 9.96921e+36f ;
inc:long_name = "Selected Incidence Angles" ;
inc:units = "degrees" ;
float lat(lat) ;
lat:_FillValue = 9.96921e+36f ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
float lon(lon) ;
lon:_FillValue = 9.96921e+36f ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;

[henry Butowsky]

You can modify the loop so the 'where' is onlu applied to type double:
you can make a list of the vars you want to operarate upon.

insead of @all=get_vars_in();
@all={"BT_3"s,"BT_4"s, "BT_H"s, "BT_V"s};

...Henry

[dominique briand]

For example, two parameters that give me negative results

byte Days_Of_Revisits(lat, lon) ;
Days_Of_Revisits:_FillValue = 0b ;
Days_Of_Revisits:long_name = "Days of revisits used for the retrieval" ;
Days_Of_Revisits:units = "NA" ;

int Mean_Acq_Time_Seconds(lat, lon) ;
Mean_Acq_Time_Seconds:_FillValue = -2147483647 ;
Mean_Acq_Time_Seconds:long_name = "Seconds of Mean time of acquisition for Soil Moisture value of DGG point" ;
Mean_Acq_Time_Seconds:units = "s" ;

[dominique briand]

Actually I process a large number of different files, and want to work with every var to detect which var contains a non 0 value!

[dominique briand]

Maybe I can explain better what I want to do, I try to detect the differences between the results of the same process over the same dataset but with different processing cluster (same software, different hardware). I first run a ncdiff, the results are sometimes really small (10^-13 for example) and I neglect them. But sometimes, the results are significative, and I noticed that isolated pixels are really different. My goal is to evealuate how many they are.

[dominique briand]

I can't fix my problem...but maybe it comes from the NaN values. One parameter just has NaN values, the result of my script gives me -32767, that corresponds to the _fillvalue...

@all=get_vars_in();
sz=@all.size();
idx=0;
print(sz);

for(idx=0;idx<sz;idx++)
{
@var_nm=@all(idx);
where(@var_nm!=0)@var_nm=1;
print(int(*@var_nm).total());
}