Hello community,
I am interested in adding mpi io support to the NCOs and wanted
to ask about existing experiences and plans. I have
already seen that mpi implementations of some operators (like mpncecat) exist,
but none of them offers parallel io support. Looking at the source code of mpncecat.c, I noticed attempts to parallelize the io:
Could you tell me why these attempts have been discontinued?
About me : I am currently studying Computer Science at the University of Hamburg
(in the research group "Scientific computing", which is located at the DKRZ)
and would like to contribute the aforementioned enhancement in the context of a Master Thesis.
Yours sincerly,
Christopher Bartz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Your interest in improving MPI functionality of NCO is welcome.
It is a tractable problem with obvious benefits to a large community.
The only reason we have not tackled MPI I/O is lack of resources.
As you can see, most NCO operators already have an MPI implementation.
They do not build "out of the box" because of bit rot.
They used to work, up to NCO version 3.2 I believe.
The basic algorithms are sound and could be retained, or if you
prefer, replaced. None utilize MPI IO.
We are rewriting all the operators to be netCDF4-compliant.
We are done with ncks, ncbo, and ncecat.
It would be best if you started putting MPI IO in those operators.
MPI'ing ncra/ncrcat/ncea, ncwa, ncpdq, and ncap2 will be more
challenging and ultimately more useful, but I recommend starting easy,
probably with ncks.
Once you have code we can test, we'd be happy to grant you write
authority to the sourceforge repository.
Best,
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello community,
I am interested in adding mpi io support to the NCOs and wanted
to ask about existing experiences and plans. I have
already seen that mpi implementations of some operators (like mpncecat) exist,
but none of them offers parallel io support. Looking at the source code of mpncecat.c, I noticed attempts to parallelize the io:
Could you tell me why these attempts have been discontinued?
About me : I am currently studying Computer Science at the University of Hamburg
(in the research group "Scientific computing", which is located at the DKRZ)
and would like to contribute the aforementioned enhancement in the context of a Master Thesis.
Yours sincerly,
Christopher Bartz
Dear Christopher,
Your interest in improving MPI functionality of NCO is welcome.
It is a tractable problem with obvious benefits to a large community.
The only reason we have not tackled MPI I/O is lack of resources.
As you can see, most NCO operators already have an MPI implementation.
They do not build "out of the box" because of bit rot.
They used to work, up to NCO version 3.2 I believe.
The basic algorithms are sound and could be retained, or if you
prefer, replaced. None utilize MPI IO.
We are rewriting all the operators to be netCDF4-compliant.
We are done with ncks, ncbo, and ncecat.
It would be best if you started putting MPI IO in those operators.
MPI'ing ncra/ncrcat/ncea, ncwa, ncpdq, and ncap2 will be more
challenging and ultimately more useful, but I recommend starting easy,
probably with ncks.
Once you have code we can test, we'd be happy to grant you write
authority to the sourceforge repository.
Best,
Charlie