Harry Mangalam wrote:
> During our benchmark tests the ncrcat test:
> ncrcat test 01: ncrcat joining 2^5 files.
>
> This benchmark test run with [--fl_fmt=netcdf4] soon runs out of both main
> mem AND swap (3.5GB & 8GB swap), effectively locking up sand.
>
> I'm not sure whether it's a memory leak or an oddly written routine > yet.
This is potentially very problematic as we do not wish to lock up our
machines while running standard benchmarks.
I hope it is a temporary bug and not a permanent feature.
Ed, I recommend trying valgrind on some test programs.
valgrind was indispensable in eliminating NCO memory errors.
> With the netcdf4 lib linked in but NOT using the --fl_fmt=netcdf4, it
> completes a little slower than the nc3 lib.
>
> lib # passed time
> -----------------------------------
> nc4 ncrcat: 1 1 657.25
> nc3 ncrcat: 1 1 523.19
Ditto. I would expect similar nc4 and nc3 throughputs on nc3 tasks.
Ed, any light you could shed would be appreciated.
Thanks,
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The benchmark data for those nc4/hdf5 tests that complete are
fairly impressive, especially at this stage. There are 2 that
fail, but I'm confident they'll be ironed out. The sizes of the
hdf5 file are troubling at small data sizes (almost 10x the
comparable netcdf3.6), but at larger sizes this seems to be
much better (but I haven't done a good analysis yet).
ncrcat test 01: ncrcat joining 2^5 files.
This benchmark test run with --fl_fmt=netcdf4 soon runs out of both main mem AND swap (3.5GB & 8GB swap), effectively locking up sand.
With the netcdf4 lib linked in but NOT using the --fl_fmt=netcdf4, it completes a little slower than the nc3 lib.
lib time
-----------------------------------
nc4 ncrcat: 1 1 657.25
nc3 ncrcat: 1 1 523.19
Archival copy of message to Ed Hill:
Hi Ed,
Harry Mangalam wrote:
> During our benchmark tests the ncrcat test:
> ncrcat test 01: ncrcat joining 2^5 files.
>
> This benchmark test run with [--fl_fmt=netcdf4] soon runs out of both main
> mem AND swap (3.5GB & 8GB swap), effectively locking up sand.
>
> I'm not sure whether it's a memory leak or an oddly written routine > yet.
This is potentially very problematic as we do not wish to lock up our
machines while running standard benchmarks.
I hope it is a temporary bug and not a permanent feature.
Ed, I recommend trying valgrind on some test programs.
valgrind was indispensable in eliminating NCO memory errors.
> With the netcdf4 lib linked in but NOT using the --fl_fmt=netcdf4, it
> completes a little slower than the nc3 lib.
>
> lib # passed time
> -----------------------------------
> nc4 ncrcat: 1 1 657.25
> nc3 ncrcat: 1 1 523.19
Ditto. I would expect similar nc4 and nc3 throughputs on nc3 tasks.
Ed, any light you could shed would be appreciated.
Thanks,
Charlie
(also mailed to Ed)
The benchmark data for those nc4/hdf5 tests that complete are
fairly impressive, especially at this stage. There are 2 that
fail, but I'm confident they'll be ironed out. The sizes of the
hdf5 file are troubling at small data sizes (almost 10x the
comparable netcdf3.6), but at larger sizes this seems to be
much better (but I haven't done a good analysis yet).
Seconds to complete (*=fastest)
-----------------------------
Test nc3.6 nc4 nc4
(classic) (hdf5)
ncbo: 26.46* 38.65 41.14
ncflint: 162.65 88.36 77.83*
ncea: 87.80* 112.63 110.38
ncecat: 156.60 157.75 143.90*
ncpdq: 402.66 399.06 376.50*
ncra: 250.61* 387.22 383.08
ncwa: 157.86 66.06 55.08*
-----------------------------
1244.64 1249.73 1187.91*
ncrcat: 550.72* 671.38 failed
ncap: 140.01 65.14* failed
-----------------------------
1935.37* 1986.25