Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

Pythonflu in parallel

2012-08-27
2013-01-14
  • Ivor Clifford
    Ivor Clifford
    2012-08-27

    Hi all,
    I thought it might interest you to know that I have managed to successfully run pythonflu-8.0 in parallel using python-2.7, openmpi-1.4.3 and OpenFOAM-1.6-ext.
    First, and most important, SetRootCase in Foam.OpenFOAM.include needs to be called while passing '-parallel' as an argument. Once this is done OpenFOAM will sort out the remaining MPI issues. If you're running a python script, this is pretty much all that needs to be done. So we can run a typical python script in parallel like follows:

    % mpirun -np 4 python script.py -parallel
    

    The python script will look something like this:

    from Foam.OpenFOAM import *
    from Foam.OpenFOAM.include import setRootCase
    from Foam.OpenFOAM.include import createTime
    from Foam.OpenFOAM.include import createMesh
    from Foam.finiteVolume import *
    import sys
    args = setRootCase(len( sys.argv ), sys.argv)
    runTime = createTime(args)
    mesh = createMesh(runTime)
    print mesh.V().gSum()
    

    It's a little trickier to run an interactive python session in parallel since we have to synchronously send interactive commands to each MPI process and display the results. IPython has support for this but I don't like the approach. Instead I came across a python module for openmpi called pyMPI (pympi.sourceforge.net) that, when imported will automatically handle this. It also uses the same MPI libraries as OpenFOAM and so the two are compatible.

    mpirun -np 4 -xterm -1 python -i
    

    This will open up 4 terminals, one for each MPI process. All MPI processes need to be run in a separate terminal so that the stdin and stdout streams are accessible. All output from the different MPI ranks will be displayed in their respective terminals. In the pythonstartup file, to link the stdin to the Rank 0 terminal simply import the mpi module.

    import ctypes
    ctypes.CDLL('libmpi.so.0', ctypes.RTLD_GLOBAL)
    import MPI
    

    Note we force the loading of libmpi.so before importing pyMPI to avoid undefined symbol errors. Add the '-parallel' argument to sys.argv and you're ready to go.

    import sys
    sys.argv.append('-parallel')
    from Foam.OpenFOAM import *
    from Foam.OpenFOAM.include import setRootCase
    from Foam.OpenFOAM.include import createTime
    from Foam.OpenFOAM.include import createMesh
    from Foam.finiteVolume import *
    args = setRootCase(len( sys.argv ), sys.argv)
    runTime = createTime(args)
    ...
    

    MPI will complain that you're calling MPI_Init twice (once from pyMPI and once from OpenFOAM) but will still work.

    I hope this will interest some pythonflu users.

    Ivor

     
  • Thanks a lot. That was very useful. I haven't tried out the interactive thing, but the other one works.