ncdiff v3.x produces some wrong differences in output. This applies to ia32 and ia64. ncdiff v2.7.1 works correctly, though. It seems to be related to the fact that the input variable "temp" is packed shorts.
Here's the command:
ncdiff -O in1.nc in2.nc out.nc
And the tiny input files:
ftp://ftp.gfdl.noaa.gov/pub/rsz/ncdiffbug/
Thanks,
Remik
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for this report.
I have confirmed that the current NCO has a problem
differencing some (but not all) packed variables.
Subtracting any variable from itself should result in zero
or missing_value's.
The following command demonstrates that this is not always
the case:
Unfortunately, the cases we use in the regression tests DO
work so we didn't notice the problem until you pointed it
out. I'm glad we have a chance to fix this in the upcoming
3.9.0 release.
Stay tuned,
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think I found and fixed the ncdiff problem you reported.
Sometime after 2.7.1, I don't when, I introduced a bug
that started writing the data buffers as their packed type
rather than their unpacked type. This bug only affected
packed data, which I don't use very much, and snuck through
the regression tests. Anyway, it's fixed in the CVS version
if you want to give that a try.
Thanks for the great bug reports!
Charlie
zender@ashes:~$ ncdiff -O -v '.?pck.?' -p ~/nco/data in.nc in.nc ~/foo.nc
zender@ashes:~$ ncks ~/foo.nc
Opened file /home/zender/foo.nc: dimensions = 2, variables = 9, global atts. = 5, id = 65536, type = NC_FORMAT_CLASSIC
Record dimension: name = time, size = 10
Global attribute 0: Conventions, size = 6 NC_CHAR, value = CF-1.0
Global attribute 1: history, size = 131 NC_CHAR, value = Thu May 24 21:38:05 2007: ncdiff -O -v .?pck.? -p /home/zender/nco/data in.nc in.nc /home/zender/foo.nc
History global attribute.
Global attribute 2: julian_day, size = 1 NC_DOUBLE, value = 200000.04
Global attribute 3: RCS_Header, size = 8 NC_CHAR, value = $Header$
Global attribute 4: nco_openmp_thread_number, size = 1 NC_INT, value = 1
ncdiff v3.x produces some wrong differences in output. This applies to ia32 and ia64. ncdiff v2.7.1 works correctly, though. It seems to be related to the fact that the input variable "temp" is packed shorts.
Here's the command:
ncdiff -O in1.nc in2.nc out.nc
And the tiny input files:
ftp://ftp.gfdl.noaa.gov/pub/rsz/ncdiffbug/
Thanks,
Remik
Hi Remik,
Thanks for this report.
I have confirmed that the current NCO has a problem
differencing some (but not all) packed variables.
Subtracting any variable from itself should result in zero
or missing_value's.
The following command demonstrates that this is not always
the case:
ncdiff -O -v '.?pck.?' -p ~/nco/data in.nc in.nc ~/foo.nc
ncks ~/foo.nc
Unfortunately, the cases we use in the regression tests DO
work so we didn't notice the problem until you pointed it
out. I'm glad we have a chance to fix this in the upcoming
3.9.0 release.
Stay tuned,
Charlie
Hi Remik,
I think I found and fixed the ncdiff problem you reported.
Sometime after 2.7.1, I don't when, I introduced a bug
that started writing the data buffers as their packed type
rather than their unpacked type. This bug only affected
packed data, which I don't use very much, and snuck through
the regression tests. Anyway, it's fixed in the CVS version
if you want to give that a try.
Thanks for the great bug reports!
Charlie
zender@ashes:~$ ncdiff -O -v '.?pck.?' -p ~/nco/data in.nc in.nc ~/foo.nc
zender@ashes:~$ ncks ~/foo.nc
Opened file /home/zender/foo.nc: dimensions = 2, variables = 9, global atts. = 5, id = 65536, type = NC_FORMAT_CLASSIC
Record dimension: name = time, size = 10
Global attribute 0: Conventions, size = 6 NC_CHAR, value = CF-1.0
Global attribute 1: history, size = 131 NC_CHAR, value = Thu May 24 21:38:05 2007: ncdiff -O -v .?pck.? -p /home/zender/nco/data in.nc in.nc /home/zender/foo.nc
History global attribute.
Global attribute 2: julian_day, size = 1 NC_DOUBLE, value = 200000.04
Global attribute 3: RCS_Header, size = 8 NC_CHAR, value = $Header$
Global attribute 4: nco_openmp_thread_number, size = 1 NC_INT, value = 1
lon: # dim. = 1, NC_FLOAT, # att. = 2, ID = 0
lon dimension 0: lon, size = 4 NC_FLOAT, dim. ID = 0 (CRD)
lon memory size is 4*nco_typ_lng(NC_FLOAT) = 4*4 = 16 bytes
lon attribute 0: long_name, size = 31 NC_CHAR, value = Longitude (typically midpoints)
lon attribute 1: units, size = 12 NC_CHAR, value = degrees_east
pck: # dim. = 0, NC_DOUBLE, # att. = 2, ID = 2
pck memory size is 1*nco_typ_lng(NC_DOUBLE) = 1*8 = 8 bytes
pck attribute 0: long_name, size = 22 NC_CHAR, value = Packed scalar variable
pck attribute 1: note, size = 209 NC_CHAR, value = Original packed value was 1s with scale_factor = 2.0d and add_offset = 1.0d. Unpacked value should be 3.0 = 2.0d*1s + 1.0d. NCO algorithms would pack this variable as scale_factor = 0.0d and add_offset = 3.0d.
pck_arr: # dim. = 1, NC_DOUBLE, # att. = 2, ID = 3
pck_arr dimension 0: lon, size = 4 NC_FLOAT, dim. ID = 0 (CRD)
pck_arr memory size is 4*nco_typ_lng(NC_DOUBLE) = 4*8 = 32 bytes
pck_arr attribute 0: long_name, size = 12 NC_CHAR, value = Packed array
pck_arr attribute 1: note, size = 67 NC_CHAR, value = Packed value is -32767s, 0s, 1s, 32767s, unpacked is same in double
rec_var_dbl_mss_val_dbl_pck: # dim. = 1, NC_DOUBLE, # att. = 3, ID = 4
rec_var_dbl_mss_val_dbl_pck dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 1 (CRD)(REC)
rec_var_dbl_mss_val_dbl_pck memory size is 10*nco_typ_lng(NC_DOUBLE) = 10*8 = 80 bytes
rec_var_dbl_mss_val_dbl_pck attribute 0: long_name, size = 59 NC_CHAR, value = record variable, double, with double missing values, packed
rec_var_dbl_mss_val_dbl_pck attribute 1: purpose, size = 45 NC_CHAR, value = Packed version of rec_var_dbl_mss_val_dbl_upk
rec_var_dbl_mss_val_dbl_pck attribute 2: missing_value, size = 1 NC_DOUBLE, value = -999
rec_var_dbl_mss_val_dbl_pck_lng: # dim. = 1, NC_DOUBLE, # att. = 3, ID = 8
rec_var_dbl_mss_val_dbl_pck_lng dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 1 (CRD)(REC)
rec_var_dbl_mss_val_dbl_pck_lng memory size is 10*nco_typ_lng(NC_DOUBLE) = 10*8 = 80 bytes
rec_var_dbl_mss_val_dbl_pck_lng attribute 0: long_name, size = 66 NC_CHAR, value = record variable, double packed as long, with double missing values
rec_var_dbl_mss_val_dbl_pck_lng attribute 1: purpose, size = 104 NC_CHAR, value = although not usual, packing doubles into longs (rather than shorts) is still a considerable cost savings
rec_var_dbl_mss_val_dbl_pck_lng attribute 2: missing_value, size = 1 NC_DOUBLE, value = -999
rec_var_dbl_pck: # dim. = 1, NC_DOUBLE, # att. = 2, ID = 7
rec_var_dbl_pck dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 1 (CRD)(REC)
rec_var_dbl_pck memory size is 10*nco_typ_lng(NC_DOUBLE) = 10*8 = 80 bytes
rec_var_dbl_pck attribute 0: long_name, size = 31 NC_CHAR, value = record variable, double, packed
rec_var_dbl_pck attribute 1: purpose, size = 71 NC_CHAR, value = Demonstrate that rounding of means of packed data are handled correctly
rec_var_flt_pck: # dim. = 1, NC_FLOAT, # att. = 2, ID = 6
rec_var_flt_pck dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 1 (CRD)(REC)
rec_var_flt_pck memory size is 10*nco_typ_lng(NC_FLOAT) = 10*4 = 40 bytes
rec_var_flt_pck attribute 0: long_name, size = 30 NC_CHAR, value = record variable, float, packed
rec_var_flt_pck attribute 1: purpose, size = 71 NC_CHAR, value = Demonstrate that rounding of means of packed data are handled correctly
scl_dbl_pck: # dim. = 0, NC_DOUBLE, # att. = 2, ID = 5
scl_dbl_pck memory size is 1*nco_typ_lng(NC_DOUBLE) = 1*8 = 8 bytes
scl_dbl_pck attribute 0: long_name, size = 31 NC_CHAR, value = scalar variable, double, packed
scl_dbl_pck attribute 1: purpose, size = 52 NC_CHAR, value = Packed version of number with ncdiff subtraction bug
time: # dim. = 1, NC_DOUBLE, # att. = 0, ID = 1
time dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 1 (CRD)(REC)
time memory size is 10*nco_typ_lng(NC_DOUBLE) = 10*8 = 80 bytes
lon[0]=0 degrees_east
lon[1]=90 degrees_east
lon[2]=180 degrees_east
lon[3]=270 degrees_east
pck = 0
lon[0]=0 pck_arr[0]=0
lon[1]=90 pck_arr[1]=0
lon[2]=180 pck_arr[2]=0
lon[3]=270 pck_arr[3]=0
time[0]=1 rec_var_dbl_mss_val_dbl_pck[0]=-999
time[1]=2 rec_var_dbl_mss_val_dbl_pck[1]=0
time[2]=3 rec_var_dbl_mss_val_dbl_pck[2]=0
time[3]=4 rec_var_dbl_mss_val_dbl_pck[3]=0
time[4]=5 rec_var_dbl_mss_val_dbl_pck[4]=0
time[5]=6 rec_var_dbl_mss_val_dbl_pck[5]=0
time[6]=7 rec_var_dbl_mss_val_dbl_pck[6]=0
time[7]=8 rec_var_dbl_mss_val_dbl_pck[7]=0
time[8]=9 rec_var_dbl_mss_val_dbl_pck[8]=-999
time[9]=10 rec_var_dbl_mss_val_dbl_pck[9]=-999
time[0]=1 rec_var_dbl_mss_val_dbl_pck_lng[0]=-999
time[1]=2 rec_var_dbl_mss_val_dbl_pck_lng[1]=0
time[2]=3 rec_var_dbl_mss_val_dbl_pck_lng[2]=0
time[3]=4 rec_var_dbl_mss_val_dbl_pck_lng[3]=0
time[4]=5 rec_var_dbl_mss_val_dbl_pck_lng[4]=0
time[5]=6 rec_var_dbl_mss_val_dbl_pck_lng[5]=0
time[6]=7 rec_var_dbl_mss_val_dbl_pck_lng[6]=0
time[7]=8 rec_var_dbl_mss_val_dbl_pck_lng[7]=0
time[8]=9 rec_var_dbl_mss_val_dbl_pck_lng[8]=-999
time[9]=10 rec_var_dbl_mss_val_dbl_pck_lng[9]=-999
time[0]=1 rec_var_dbl_pck[0]=0
time[1]=2 rec_var_dbl_pck[1]=0
time[2]=3 rec_var_dbl_pck[2]=0
time[3]=4 rec_var_dbl_pck[3]=0
time[4]=5 rec_var_dbl_pck[4]=0
time[5]=6 rec_var_dbl_pck[5]=0
time[6]=7 rec_var_dbl_pck[6]=0
time[7]=8 rec_var_dbl_pck[7]=0
time[8]=9 rec_var_dbl_pck[8]=0
time[9]=10 rec_var_dbl_pck[9]=0
time[0]=1 rec_var_flt_pck[0]=0
time[1]=2 rec_var_flt_pck[1]=0
time[2]=3 rec_var_flt_pck[2]=0
time[3]=4 rec_var_flt_pck[3]=0
time[4]=5 rec_var_flt_pck[4]=0
time[5]=6 rec_var_flt_pck[5]=0
time[6]=7 rec_var_flt_pck[6]=0
time[7]=8 rec_var_flt_pck[7]=0
time[8]=9 rec_var_flt_pck[8]=0
time[9]=10 rec_var_flt_pck[9]=0
scl_dbl_pck = 0
time[0]=1
time[1]=2
time[2]=3
time[3]=4
time[4]=5
time[5]=6
time[6]=7
time[7]=8
time[8]=9
time[9]=10
I'll try the updated cvs. Thank you.
Remik