NCO 4.1.0, just released last week, includes the MM3 patches described as "done" above.
You can install from scratch or copy the executables from their usual locations in ~zender/bin.
Regarding Andy Mai and nc__open() instead of nc_open():
I've never tried it. Russ, is there reason to expect this to work? on which architectures?
is there a downside to always using nc__open() with NC_SIZEHINT_DEFAULT?
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I haven't tried it either, but I don't think just changing nc_open to nc__open (or nc_create to nc__create) with a large value for the bufrsizehint argument will only speed things up if the bufrsizehint was large enough to span several records, and even then the speedup would be less than processing each record completely for all record variables, as you are doing now. In the extreme, if bufrsizehint were big enough to contain all the file's records, it would bring everything into memory on the first record access and might achieve the desired speedup, but that's typically way more memory than is available or than is used for the record-at-a-time algorithm.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks Russ for your comments. Based on those, I'll stick with the
original plan above to complete patching of NCO for MM3 slowness.
This will take some time because in many operators it requires
significant re-architecting.
At the same time, adding a nc__open()/nc__create() option to all
NCO operators is certainly possible so that is now TODO nco1041.
It might help power users optimize production uses, so no harm (knock
on wood) in making it available, too.
Si, I recommend making NCO 4.1.0 available to users as the most
immediate and best-tested MM3 workaround for NCO.
c
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Just as a heads-up, the current netCDF snapshot has a "diskless netCDF" feature that will be available in the next release. By calling nc_create() with the NC_DISKLESS flag set, a memory-resident file can be created without disk I/O. On close, it can be optionally persisted into a real disk file. This works for both netCDF-3 and netCDF-4 files, but might offer an alternative approach, when writing an MM3 file, to speed things up. I hope to run some benchmarks using NC_DISKLESS soon …
-Russ
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Si,
NCO 4.1.0, just released last week, includes the MM3 patches described as "done" above.
You can install from scratch or copy the executables from their usual locations in ~zender/bin.
Regarding Andy Mai and nc__open() instead of nc_open():
I've never tried it. Russ, is there reason to expect this to work? on which architectures?
is there a downside to always using nc__open() with NC_SIZEHINT_DEFAULT?
cz
I haven't tried it either, but I don't think just changing nc_open to nc__open (or nc_create to nc__create) with a large value for the bufrsizehint argument will only speed things up if the bufrsizehint was large enough to span several records, and even then the speedup would be less than processing each record completely for all record variables, as you are doing now. In the extreme, if bufrsizehint were big enough to contain all the file's records, it would bring everything into memory on the first record access and might achieve the desired speedup, but that's typically way more memory than is available or than is used for the record-at-a-time algorithm.
Thanks Russ for your comments. Based on those, I'll stick with the
original plan above to complete patching of NCO for MM3 slowness.
This will take some time because in many operators it requires
significant re-architecting.
At the same time, adding a nc__open()/nc__create() option to all
NCO operators is certainly possible so that is now TODO nco1041.
It might help power users optimize production uses, so no harm (knock
on wood) in making it available, too.
Si, I recommend making NCO 4.1.0 available to users as the most
immediate and best-tested MM3 workaround for NCO.
c
Just as a heads-up, the current netCDF snapshot has a "diskless netCDF" feature that will be available in the next release. By calling nc_create() with the NC_DISKLESS flag set, a memory-resident file can be created without disk I/O. On close, it can be optionally persisted into a real disk file. This works for both netCDF-3 and netCDF-4 files, but might offer an alternative approach, when writing an MM3 file, to speed things up. I hope to run some benchmarks using NC_DISKLESS soon …
-Russ