ncks slowdown from 4.3.4 to 4.4.3

2014-05-07
2014-05-07
  • All,

    I'm experiencing a slowdown in ncks performance from version 4.3.4 to 4.4.3 and wondering if this is known and if a solution is available.

    To start, I maintain a set of base libraries for the GEOS-5 GCM and included in them is NCO. Recently, I decided to upgrade as many of them as I could to "most recent stable". Among them was NCO which went from NCO 4.3.4 to 4.4.3. I did this mainly because there had been bug fixes in 4.4.3 some of which Charlie had fixed.

    However, in doing so, I noticed that a sample test script I like to run on each upgrade got slower. The call I do is (or was when I made the test) a typical post processing task:

    ncks --hst --abc -O -4 -L 2 --cnk_plc=g2d --cnk_dmn lat,91 \
        --cnk_dmn lon,144 --cnk_dmn lev,1 \
        TestFiles/e5110_fp.inst3_3d_asm_Cp.20131001_09z+20131001_1200z.nc4 \
        NewNCO/e5110_fp.inst3_3d_asm_Cp.20131001_09z+20131001_1200z.nc4
    

    where the testfile is a 326 MB NC4 file from a run.

    When I run this with NCO 4.3.4 it takes 13 seconds (mean over 5 runs using /usr/bin/time). With NCO 4.4.3, it takes 19.29 seconds.

    To be careful, the test I did was to take the libraries that use NCO 4.4.3 and just substituted NCO 4.3.4 and built that. There is no difference between how I compile NCO or anything. They use the same zlib, szlib, hdf4, hdf5, netcdf, etc. as well as the same compiler.

    Note, both versions output the same file as far as "cdo diffn" is concerned, so that is good, it's just ~30-40% slower. And as there is the possibility of many, many of these calls being done, it'd be nice to regain that speed.

    So, is there, perhaps, a flag I could pass to ncks-4.4.3 to regain that speed? Or maybe I need to build NCO 4.4.3 with a different --enable/--disable flag?

    Thanks,
    Matt Thompson

     
  • Charlie Zender
    Charlie Zender
    2014-05-07

    Hi Matt,
    I'm unsure what caused the slowdown. If anyone has suggestions, please post them.
    cz