Regarding parallel io support in nco

  • Christopher

    Hello community,
    I am interested in adding mpi io support to the NCOs and wanted
    to ask about existing experiences and plans. I have
    already seen that mpi implementations of some operators (like mpncecat) exist,
    but none of them offers parallel io support. Looking at the source code of mpncecat.c, I noticed attempts to parallelize the io:

    #if 0
        /* fxm: netCDF4: Change to independent variable reads? */
    #ifdef ENABLE_NETCDF4
    #endif /* !ENABLE_NETCDF4 */
    #endif /* !ENABLE_PNETCDF */
    #endif /* !0 */

    Could you tell me why these attempts have been discontinued?

    About me : I am currently studying Computer Science at the University of Hamburg
    (in the research group "Scientific computing", which is located at the DKRZ)
    and would like to contribute the aforementioned enhancement in the context of a Master Thesis.

    Yours sincerly,

    Christopher Bartz

  • Charlie Zender
    Charlie Zender

    Dear Christopher,

    Your interest in improving MPI functionality of NCO is welcome.
    It is a tractable problem with obvious benefits to a large community.
    The only reason we have not tackled MPI I/O is lack of resources.

    As you can see, most NCO operators already have an MPI implementation.
    They do not build "out of the box" because of bit rot.
    They used to work, up to NCO version 3.2 I believe.
    The basic algorithms are sound and could be retained, or if you
    prefer, replaced. None utilize MPI IO.

    We are rewriting all the operators to be netCDF4-compliant.
    We are done with ncks, ncbo, and ncecat.
    It would be best if you started putting MPI IO in those operators. 

    MPI'ing ncra/ncrcat/ncea, ncwa, ncpdq, and ncap2 will be more
    challenging and ultimately more useful, but I recommend starting easy,
    probably with ncks.
    Once you have code we can test, we'd be happy to grant you write
    authority to the sourceforge repository.