option to add md5 to local/global attr

ocehugo
2013-07-23
2013-10-17
  • ocehugo
    ocehugo
    2013-07-23

    Hello,

    I would like to request this option for ncks in particular:

    -add_md5_vars_att

    This would add the md5 checksum to the variable attributes, or a global attribute of the checked variables.

    Why:

    Since this feature have appears, i have been using it for all my operations that request data over opendap servers. But searching for a bug in my workflow, i have found some files with strange values in some variables requested on an odap server. The md5 should help me avoid the wrong data issues, but it not work for those particular requests (4 of ~4000 requests, until now).  Since the md5 output is only show when requesting debug information and i don't have the full log of my operations, it will be important to store the md5 for each variable in a fashion way, so when this kind of problem occurs, i can check the md5 between the wrong variable and the md5 of a new request and check it ( and see if the opendap server send me a wrong dataset or whas NCO and it's operations that have changed the dataset).

    Also, this would avoid storing all the output of a command with -D option.

    I thing this should be inside the variable att, something like md5_checksum_value or something alike, since it can be easily readed.

    PS: This problem was very tricky, because in some other files with wrong values the values are absurd (like missing values around some "good" values), but in this particular case the values was inside an acceptable global range, but with range not representing the region/location ( like air temperature of earth surface located in the high atmosphere). It's very strange since it have never happen before, so i don't know if was the opendap-server or NCO that have cause this.

     
  • Charlie Zender
    Charlie Zender
    2013-07-23

    Thanks for your feedback.
    Yes, this would be a reasonable feature. It was already in the nco TODO list as nco1034 for ncatted, and I have added nco1101 for ncks. it should probably go into ncks first. I'll post here if/when there is more news on this.
    cz

     
  • Charlie Zender
    Charlie Zender
    2013-07-24

    the repository version of ncks now writes MD5 attributes to netCDF4 files when invoked with -md5 -D 6 (or higher).
    an example command to MD5 a file is
    ncks -O -C -4 -md5 -D 6 -v md5_.? ~/nco/data/in.nc ~/foo.nc
    writing to netCDF3 files is do-able, but i havn't yet decided how to implement that in the released version.
    please test and post feedback.
    cz

     
  • Charlie Zender
    Charlie Zender
    2013-07-25

    This now works with netCDF3 files too, and is invoked with the -md5_wrt switch, e.g.,
    ncks -md5_wrt ~/nco/data/in.nc ~/foo.nc
    http://nco.sf.net/nco.html#md5