I'm experiencing a slowdown in ncks performance from version 4.3.4 to 4.4.3 and wondering if this is known and if a solution is available.
To start, I maintain a set of base libraries for the GEOS-5 GCM and included in them is NCO. Recently, I decided to upgrade as many of them as I could to "most recent stable". Among them was NCO which went from NCO 4.3.4 to 4.4.3. I did this mainly because there had been bug fixes in 4.4.3 some of which Charlie had fixed.
However, in doing so, I noticed that a sample test script I like to run on each upgrade got slower. The call I do is (or was when I made the test) a typical post processing task:
where the testfile is a 326 MB NC4 file from a run.
When I run this with NCO 4.3.4 it takes 13 seconds (mean over 5 runs using /usr/bin/time). With NCO 4.4.3, it takes 19.29 seconds.
To be careful, the test I did was to take the libraries that use NCO 4.4.3 and just substituted NCO 4.3.4 and built that. There is no difference between how I compile NCO or anything. They use the same zlib, szlib, hdf4, hdf5, netcdf, etc. as well as the same compiler.
Note, both versions output the same file as far as "cdo diffn" is concerned, so that is good, it's just ~30-40% slower. And as there is the possibility of many, many of these calls being done, it'd be nice to regain that speed.
So, is there, perhaps, a flag I could pass to ncks-4.4.3 to regain that speed? Or maybe I need to build NCO 4.4.3 with a different --enable/--disable flag?
Thanks,
Matt Thompson
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It looks like NCO 4.4.9 has fixed this performance regression for me. Here's 4.4.8 timings:
user system
Min. :17.02 Min. :0.230
1st Qu.:17.21 1st Qu.:0.240
Median :17.39 Median :0.240
Mean :17.37 Mean :0.248
3rd Qu.:17.62 3rd Qu.:0.260
Max. :17.63 Max. :0.270
and here's 4.4.9:
user system
Min. : 9.36 Min. :0.20
1st Qu.:11.16 1st Qu.:0.22
Median :11.64 Median :0.24
Mean :11.26 Mean :0.23
3rd Qu.:12.06 3rd Qu.:0.24
Max. :12.08 Max. :0.25
I'll take that speed up.
Thanks,
Matt
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you, Matthew, for checking these timings and posting them. The only change from 4.4.8 to 4.4.9 that could explain the speed-up is chunking improvements. The good thing about this speed up is that it might be two-way: Faster to write may mean faster to read, though that remains to be seen...
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
All,
I'm experiencing a slowdown in ncks performance from version 4.3.4 to 4.4.3 and wondering if this is known and if a solution is available.
To start, I maintain a set of base libraries for the GEOS-5 GCM and included in them is NCO. Recently, I decided to upgrade as many of them as I could to "most recent stable". Among them was NCO which went from NCO 4.3.4 to 4.4.3. I did this mainly because there had been bug fixes in 4.4.3 some of which Charlie had fixed.
However, in doing so, I noticed that a sample test script I like to run on each upgrade got slower. The call I do is (or was when I made the test) a typical post processing task:
ncks --hst --abc -O -4 -L 2 --cnk_plc=g2d --cnk_dmn lat,91 \ --cnk_dmn lon,144 --cnk_dmn lev,1 \ TestFiles/e5110_fp.inst3_3d_asm_Cp.20131001_09z+20131001_1200z.nc4 \ NewNCO/e5110_fp.inst3_3d_asm_Cp.20131001_09z+20131001_1200z.nc4where the testfile is a 326 MB NC4 file from a run.
When I run this with NCO 4.3.4 it takes 13 seconds (mean over 5 runs using /usr/bin/time). With NCO 4.4.3, it takes 19.29 seconds.
To be careful, the test I did was to take the libraries that use NCO 4.4.3 and just substituted NCO 4.3.4 and built that. There is no difference between how I compile NCO or anything. They use the same zlib, szlib, hdf4, hdf5, netcdf, etc. as well as the same compiler.
Note, both versions output the same file as far as "cdo diffn" is concerned, so that is good, it's just ~30-40% slower. And as there is the possibility of many, many of these calls being done, it'd be nice to regain that speed.
So, is there, perhaps, a flag I could pass to ncks-4.4.3 to regain that speed? Or maybe I need to build NCO 4.4.3 with a different --enable/--disable flag?
Thanks,
Matt Thompson
Hi Matt,
I'm unsure what caused the slowdown. If anyone has suggestions, please post them.
cz
It looks like NCO 4.4.9 has fixed this performance regression for me. Here's 4.4.8 timings:
and here's 4.4.9:
I'll take that speed up.
Thanks,
Matt
Thank you, Matthew, for checking these timings and posting them. The only change from 4.4.8 to 4.4.9 that could explain the speed-up is chunking improvements. The good thing about this speed up is that it might be two-way: Faster to write may mean faster to read, though that remains to be seen...
cz