#305 foam-extend 3.2 w intel mpi 5.1.3.210

[mpantic]

Hello,

We are trying to compile and run foam-extend 3.2 on a linux cluster with Intel MPI. Compilation finishes successfully using the options:
/etc/prefs.sh
export WM_COMPILER=Icc
export WM_MPLIB=INTELMPI

Serial runs do work in general without any problem. For simple cases such as the cavity example (icoFoam), parallel runs also work using mpirun and the command line switch -parallel, regardless of the number of processes (with correct decomposition) or number of nodes.

For more complex examples/meshes, such as the motorbike example (simpleFoam) or custom cases (pimpleDyMFoam) the system behaves as follows:
Serial runs work
Parallel runs with exactly 2 processes work using Intel MPI, also over 2 nodes.
* Parallel runs with more than 2 processes using Intel MPI crash with a segfault, regardless of whether they run on the same machine or different machines

I tried running a case with 3 nodes using mpirun and gdb, with the following output:
1] Program received signal SIGSEGV, Segmentation fault.
[1] 0x00007f71fc723579 in Foam::processorFvPatchField<Foam::Tensor<double> >::initEvaluate(Foam::Pstream::commsTypes) ()

As this seems to be related to the Pstream library and the set up of the interprocess com, I also tried to switch commsTypes. The behaviour is the same, regardless of commsType (tested blocking, nonBlocking and scheduled).

It does not seem to be a problem with Intel MPI, as other software (e.g. Openfoam 2.3-x) , some cases with foam-extend 3.2 and all cases with foam-extend 3.2 with only 2 processes run correctly using Intel MPI and the underlying infiniband stack. All MPI variables such as MPI_ROOT are set correctly.

OS is CentOS 7.1.1503 w/Kernel 3.10.0-229.20.1.el7.x86_64
ICC is version 16.0.3 20160415
MPI is version Intel(R) MPI Library for Linux OS, Version 5.1.3 Build 20160120 (build id: 14053)
Code is checked out from git://git.code.sf.net/p/foam-extend/foam-extend-3.2, latest history entry is 334ba0562a2cf50a1c700d240d85f9e07c8175b4/ Final cumulative bug fix for 3.2. Author: Hrvoje Jasak.

Please let me know if there is any known bug (haven't found anything) or any further things/patches/branches I could test.

Thank you very much for your help!

[mpantic]

Title text ended up in "Labels" field.. Please adjust title to: "Segfault with foam-extend 3.2 and Intel MPI", thanks!