Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#49 ncpdq introduces bias to packed data

closed-accepted
None
5
2013-09-08
2011-10-20
Neil Davis
No

I found that ncpdq introduces a bias when packing data. I was able to eliminate the bias by modifying the code nco_cnf_typ.c to add math.h and round the data on lines 333 and 334 as shown below.

case NC_FLOAT: for(idx=0L;idx<sz;idx++) {val_out.sp[idx]=(short int)lroundf(val_in.fp[idx]);} break; /* Coerce to avoid C++ compiler assignment warning */
case NC_DOUBLE: for(idx=0L;idx<sz;idx++) {val_out.sp[idx]=(short int)lround(val_in.dp[idx]);} break; /* Coerce to avoid C++ compiler assignment warning */

I tested this and the bias was removed with this method and it is the method recommended by NetCDF best practices, however I'm not sure what impact this will have on other functions which use that routine. One potential fix would be to round the data before passing it to the conforming function. I wasn't sure how to go about doing that however and didn't know if there were other pitfalls.

I'm going to upload images of the biased and non-biased packing

Discussion

  • Neil Davis
    Neil Davis
    2011-10-20

    Packed vs non-packed with bias

     
    Attachments
  • Neil Davis
    Neil Davis
    2011-10-20

    Packed vs non-packed without bias

     
    Attachments
  • Charlie Zender
    Charlie Zender
    2011-12-07

    • assigned_to: nobody --> zender
    • status: open --> pending-accepted
     
  • Charlie Zender
    Charlie Zender
    2011-12-07

    Thank you for reporting this. I implemented versions of your suggested patch in the latest code. It does produce additional regressions with other functions. I am trying to track those down. I expect this will be fixed in 4.0.9. If you have time to try 4.0.9 beta now and tell me if it works for you that would be helpful.
    Charlie

     
  • Neil Davis
    Neil Davis
    2011-12-12

    I'd be happy to try 4.0.9 beta, which version of the source should I download?

    Neil

     
  • Neil Davis
    Neil Davis
    2011-12-12

    • status: pending-accepted --> open-accepted
     
  • Charlie Zender
    Charlie Zender
    2011-12-12

    by 4.0.9 beta, i simply mean the latest snapshot, i.e., the main trunk of code that you can check out with cvs as described on the nco homepage. thanks. cz

     
  • Neil Davis
    Neil Davis
    2011-12-13

    I have checked out the lastest version of the source.

    I am working on getting it compiled and have found a few errors including one I'm not sure how to get around.

    ./src/nco/nco_lst_utl.c Need to add #include <regex.h> after line #ifdef NCO_HAVE_REGEX_FUNCTIONALITY

    ./src/nco/nco_mmr.h had to remove IFDEF commands around line # include <sys/resource.h> /* Resource usage and limits */

    The file ./src/nco/nco_grp_utl.c seems to require nco_netcdf.c to have been compiled previously however it is not. This is where I get stuck in the compilation.

    Thanks,
    Neil

     
  • Charlie Zender
    Charlie Zender
    2011-12-13

    oops. that build problem is my fault. i will post again by the weekend when i have had a chance to fix it. sorry.cz

     
  • Charlie Zender
    Charlie Zender
    2013-09-08

    This was fixed in version 4.0.9, then that fix was changed for standards compliance in 4.3.0.

     
  • Charlie Zender
    Charlie Zender
    2013-09-08

    • status: open-accepted --> closed-accepted