Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Regarding parallel io support in nco

Developers
2013-04-30
2013-10-17
  • Christopher
    Christopher
    2013-04-30

    Hello community,
    I am interested in adding mpi io support to the NCOs and wanted
    to ask about existing experiences and plans. I have
    already seen that mpi implementations of some operators (like mpncecat) exist,
    but none of them offers parallel io support. Looking at the source code of mpncecat.c, I noticed attempts to parallelize the io:

    #if 0
        /* fxm: netCDF4: Change to independent variable reads? */
    #ifdef ENABLE_NETCDF4
        rcd=nco_open_par(fl_in,NC_MPIIO|NC_NETCDF4,mpi_cmm,mpi_nfo,&in_id);
    #endif /* !ENABLE_NETCDF4 */
    #ifdef ENABLE_PNETCDF
        rcd=ncompi_open(mpi_cmm,fl_in,NC_NOWRITE,mpi_nfo,&in_id);
    #endif /* !ENABLE_PNETCDF */
    #endif /* !0 */
    

    Could you tell me why these attempts have been discontinued?

    About me : I am currently studying Computer Science at the University of Hamburg
    (in the research group "Scientific computing", which is located at the DKRZ)
    and would like to contribute the aforementioned enhancement in the context of a Master Thesis.

    Yours sincerly,

    Christopher Bartz

     
  • Charlie Zender
    Charlie Zender
    2013-04-30

    Dear Christopher,

    Your interest in improving MPI functionality of NCO is welcome.
    It is a tractable problem with obvious benefits to a large community.
    The only reason we have not tackled MPI I/O is lack of resources.

    As you can see, most NCO operators already have an MPI implementation.
    They do not build "out of the box" because of bit rot.
    They used to work, up to NCO version 3.2 I believe.
    The basic algorithms are sound and could be retained, or if you
    prefer, replaced. None utilize MPI IO.

    We are rewriting all the operators to be netCDF4-compliant.
    We are done with ncks, ncbo, and ncecat.
    It would be best if you started putting MPI IO in those operators. 

    MPI'ing ncra/ncrcat/ncea, ncwa, ncpdq, and ncap2 will be more
    challenging and ultimately more useful, but I recommend starting easy,
    probably with ncks.
    Once you have code we can test, we'd be happy to grant you write
    authority to the sourceforge repository.

    Best,
    Charlie