Pythonflu in parallel

[Ivor Clifford]

Hi all,
I thought it might interest you to know that I have managed to successfully run pythonflu-8.0 in parallel using python-2.7, openmpi-1.4.3 and OpenFOAM-1.6-ext.
First, and most important, SetRootCase in Foam.OpenFOAM.include needs to be called while passing '-parallel' as an argument. Once this is done OpenFOAM will sort out the remaining MPI issues. If you're running a python script, this is pretty much all that needs to be done. So we can run a typical python script in parallel like follows:

% mpirun -np 4 python script.py -parallel

The python script will look something like this:

from Foam.OpenFOAM import *
from Foam.OpenFOAM.include import setRootCase
from Foam.OpenFOAM.include import createTime
from Foam.OpenFOAM.include import createMesh
from Foam.finiteVolume import *
import sys
args = setRootCase(len( sys.argv ), sys.argv)
runTime = createTime(args)
mesh = createMesh(runTime)
print mesh.V().gSum()

It's a little trickier to run an interactive python session in parallel since we have to synchronously send interactive commands to each MPI process and display the results. IPython has support for this but I don't like the approach. Instead I came across a python module for openmpi called pyMPI (pympi.sourceforge.net) that, when imported will automatically handle this. It also uses the same MPI libraries as OpenFOAM and so the two are compatible.

mpirun -np 4 -xterm -1 python -i

This will open up 4 terminals, one for each MPI process. All MPI processes need to be run in a separate terminal so that the stdin and stdout streams are accessible. All output from the different MPI ranks will be displayed in their respective terminals. In the pythonstartup file, to link the stdin to the Rank 0 terminal simply import the mpi module.

import ctypes
ctypes.CDLL('libmpi.so.0', ctypes.RTLD_GLOBAL)
import MPI

Note we force the loading of libmpi.so before importing pyMPI to avoid undefined symbol errors. Add the '-parallel' argument to sys.argv and you're ready to go.

import sys
sys.argv.append('-parallel')
from Foam.OpenFOAM import *
from Foam.OpenFOAM.include import setRootCase
from Foam.OpenFOAM.include import createTime
from Foam.OpenFOAM.include import createMesh
from Foam.finiteVolume import *
args = setRootCase(len( sys.argv ), sys.argv)
runTime = createTime(args)
...

MPI will complain that you're calling MPI_Init twice (once from pyMPI and once from OpenFOAM) but will still work.

I hope this will interest some pythonflu users.

Ivor

[Ganesh Vijayakumar]

Thanks a lot. That was very useful. I haven't tried out the interactive thing, but the other one works.