#117 Failure on an .nc file that has _NCProperties attribute

[Rostislav]

$ ncks --version
NCO netCDF Operators version 4.7.2 built by buildd on lcy01-amd64-004 at Jan 25 2018 19:04:04
ncks version 4.7.2

$ ncks -4 in.nc in.nc4
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_copy_att()
nco_err_exit(): ERROR Error code is -42. Translation into English with nc_strerror(-42) is "NetCDF: String match to name in use"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

The file results from quite complex cdo/nco/thredds combination.. The issue is that it has _NCproperties attribute. After a small workaround
$ ncatted -a _NCProperties,global,d,, in.nc
ncks passes. Such situation, probably, could have been handled more intelligently.
At least, more informative error message about which attibute has failed would be of great help...

Thank you!

[henry Butowsky]

HI Rostisfalv,
have managed to recreate your situation on my machine.
The odd thing is that the attribue NCProperties is not visible in ncdump but visible in ncks
Will investigate further

...Henry

[Charlie Zender]

Thanks for reporting this. I reproduced this issue with the latest NCO (4.9.0). I agree with your suggestion to print the attribute name before exiting on a failed write. It shouldn't be too hard to implement. I will post again when I know more.

[Charlie Zender]

I just pushed a fix to the current snapshot that will become 4.9.1. It prints the names of everything when there is an attribute copying error:

zender@sastrugi:~/nco$ ncks -O -4 ~/in.nc ~/in.nc4
ERROR: nco_copy_att() unable to copy attribute "_NCProperties" from "/" to "/" because the destination group already has an attribute of that name
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_copy_att()
nco_err_exit(): ERROR Error code is -42. Translation into English with nc_strerror(-42) is "NetCDF: String match to name in use"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

If this does not work for you, please re-open the issue.

[Charlie Zender]

I should add that all I fixed was the verbosity of the failure. _NCProperties is one of those new hidden global attributes that the netCDF library is supposed to take care of. NCO (tries to) copy files using only the documented API to copy the public attributes of the file. The fact that the original command fails is worrisome and suggest that NCO is trying to copy system attributes (rather than setting them through the public API). I have never see this problem before and I am going to ignore it unless/until someone convinces me that the error is in the NCO layer and not in the netCDF layer. _NCProperties should automatically be set at file creation by the netCDF layer, and it should not be copied by NCO.

[Rostislav]

Thank you! Your approach makes full sense to me.

I wonder if a NetCDF3 file with '_NCProperties' attribute set as a real attribute is a legitimate NetCDF.

Meanwhile I have posted the issue here: https://github.com/Unidata/netcdf-c/issues/1572 Let us see what netcdf folks say..

[Dave Allured]

Henry, you said this attribute is not visible in ncdump. However, ncdump 4.7.3 shows this attribute for me. Here is a truncated selection of the global attributes of in.nc:

// global attributes:
:CDI = "Climate Data Interface version 1.9.3 (http://mpimet.mpg.de/cdi)" ;
:Conventions = "CF-1.4" ;
:source = "SILAM v5_6 (r584445)" ;
:title = "SILAM_OUTPUT" ;
:NCO = "4.7.2" ;
:CDO = "Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)" ;
:_NCProperties = "version=1|netcdflibversion=4.4.1.1|hdf5libversion=1.10.0" ;
:_Format = "64-bit offset" ;

[Dave Allured]

My understanding is that _NCProperties is reserved for use by the netcdf library only. However, experiments with netcdf-c 4.7.3 find that the user is NOT currently prevented from writing this attribute into a netcdf-3 file, either classic or 64-bit offset.

Therefore it seems that some part of the user's original process copied an illegitimate _NCProperties attribute into a netcdf-3 file. Recent versions of the netcdf library do not prevent this.

It is an open question whether this file as a whole is considered a "legitimate" netcdf-3 file.

However, since this is happening for real in arbitrary user files, my opinion is that the desirable NCO behavior should be to explicitly detect and strip out any _NCProperties global attributes, and never copy them. Whether a warning should be printed, I'm not sure.

[Charlie Zender]

Wouldn't it be more appropriate for the netCDF library to prevent writing certain attributes (anything beginning with an underscore except _FillValue?) through the public API? Perhaps that ship has sailed (since older libraries support it) so now it is up to applications to prevent mis-use. Seems unsustainable to me.

[Dave Allured]

That is a good question. For many years, netcdf documentation has clearly stated that attribute names beginning with underscore are reserved for the library only. However, the API has consistently never enforced this, as far as I can recall. This presents a dilema for NCO and other general purpose software.

There are a few cases in the wild where custom underscore attributes have crept in. For no better reason than to avoid rekindling old debates, I think the most conservative approach for NCO would be to intercept only actual problem attributes. Read that as "make your life easier". ;-)

Therefore I suggest that intercepting problem attributes like _NCProperties, case by case only, is the easiest way forward.

[Charlie Zender]

I aggree with this approach and just surgically prevented NCO from trying to copy this attribute. The new behavior is:

zender@sastrugi:~/nco$ ncks -O -4 ~/in.nc ~/in.nc4
WARNING: nco_copy_att() requested to copy attribute "_NCProperties". As of version 4.9.1 (TBD, 2020), NCO refuses to copy this attribute as its value is exclusively reserved for the netCDF library to manipulate internally. This attribute was likely exposed to NCO via an upstream violation of the netCDF API. The netCDF library should though does not enforce this proscription itself, at least through netCDF verstion 4.7.3. To reduce headaches, NCO will proceed as though this operation were never requested.

[Dave Allured]

I had to rethink this. From a user perspective, I think these two cases should be treated the same way when the source file contains _NCProperties:

  copy netcdf-3 --> netcdf-4
  copy netcdf-4 --> netcdf-4

Certainly the second case changes the _NCProperties attribute all the time these days, without complaining. However, rogue _NCProperties in netcdf-3 files are now escaping into the wild, and we sort of understand how this happens. Printing the warning would be likely to spawn unnecessary help tickets to both NCO and netcdf. Therefore I think it would be better to quietly intercept _NCProperties, and print no warning at all.

There is an annoying quirk in recent netcdf library versions. In a netcdf-4 file, _NCProperties is treated as a hidden attribute. It is deliberately omitted from the returned set of global attributes. Ref. netcdf users guide, Appendix A, under Provenance Attributes.

However in the case of a netcdf-3 file, the behavior is undocumented. _NCProperties is then treated like an ordinary visible user attribute, and included in the returned set of global attributes.

If it were not for this inconsistency between formats, this error condition for NCO when copying files would have never occurred. So by suppressing a warning, NCO is simply covering up for this pecular behavior. I think this is okay.

[Charlie Zender]

How about INFO instead of WARNING with a worry-free ending?

zender@sastrugi:~/nco$ ncks -O -4 ~/in.nc ~/in.nc4
INFO: nco_copy_att() requested to copy attribute "_NCProperties". As of version 4.9.1 (TBD, 2020), NCO refuses to copy this attribute as its value is exclusively reserved for the netCDF library to manipulate internally. This attribute was likely exposed to NCO via an upstream violation of the netCDF API. The netCDF library should though does not enforce this proscription itself, at least through netCDF verstion 4.7.3. To reduce headaches, NCO will proceed as though this operation were never requested. This message may be safely ignored.

[Dave Allured]

Okay, that is a little better. However my preference remains to engage in an explicit cover-up ;-) , and suppress the message in the selfish hope that I will not have to answer future questions about this. At least, not in the NCO context. I leave the final choice up to you.

[Dave Allured]

A couple more tests found that Unidata's own nccopy tool (recent version) always quietly ignores any incoming _NCProperties global attribute, regardless of the source file format. This corresponds with my suggestion for NCO.

[Rostislav]

Thank you for the great tool and fantastic support! The solution seems to be best possible under the circumstances..