#251 Parallel issues with BCs (gMax, reduce...)

[Pablo Higuera]

Dear all,

after releasing IHFOAM I realized that there is some trouble running in parallel, as MPI thinks it runs out of buffer memory (MPI_ERR_TRUNCATE: message truncated). This is not the case, as MPI_BUFFER_SIZE is 20000000, the same as for other versions in which no problem arises.

I then narrowed down the problem and discovered that it is triggered when calling parallel functions: gMax, gMin, reduce...

However, this problem is not always present, it is dependent on which field the boundary condition is applied to. The same BC applied to pressure works, but when applied to alpha1 it crashes.

I have created a small example that reproduces the problem:
- customZeroGrad: a custom zero gradient BC with some parallel operations and Info statements. (Sorry for the vast amount of defined-but-unused variables, as I created the BC from the wave generation one).
- cavity_new: a very simple case derived from the cavity tutorial, ready to run with icoFoam and interFoam

The way to reproduce is as follows:
- Compile customZeroGrad:

cd customZeroGrad
./localMake

• Run the case

cd cavity_new
./runParallelIcoFoam
./cleanCase
./runParallelInterFoam_OK
./cleanCase
./runParallelInterFoam_FAIL

The first case runs without problems. The BC is applied to the p field.
The second case is also OK. The BC is applied to pd.
The third case fails. The BC is applied to alpha1. The error message is:

[user:PID] *** An error occurred in MPI_Recv
[user:PID] *** on communicator MPI_COMM_WORLD
[user:PID] *** MPI_ERR_TRUNCATE: message truncated
[user:PID] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort

There are actually two bugs here.

---

First, when the case runs, gMin and gMax do not work as expected. The minimum and maximum points of the patch are calculated in two different ways: with a loop and a reduce statement, and the result is right (minX = 0.1 and maxX = 0.2).

And then with gMin and gMax a warning appears:

--> FOAM Warning : 
    From function max(const UList<Type>&)
    in file ~/foam/foam-extend-3.1/src/foam/lnInclude/FieldFunctions.C at line 321
    empty field, returning zero

and the results are wrong:

GM Procesador 1: xMin 0, xMax 0.2
GM Procesador 0: xMin 0, xMax 0.2
GM Procesador 2: xMin 0, xMax 0.2
GM Procesador 3: xMin 0, xMax 0.2

maximums are calculated fine for all the processors and minimums are not.

---

The second bug is the failure when the parallel functions are called for the alpha1 field, as they have proven to work elsewhere...

Best regards,

Pablo

[Bernhard Gschaider]

Only had a quick look. Could it be that this is the same problem as http://sourceforge.net/p/openfoam-extend/ticketsswak4foam/123/

Try the described workaround with setting the commsType

[Pablo Higuera]

Thank you Bernhard, it works! Does the nonBlocking type have any further implications?

Regarding the first bug, I now see the point you made here:
http://www.cfd-online.com/Forums/openfoam-bugs/66024-inconsistent-behaviour-gmax-gmin-parallel-runs.html

0 is streamed over all the processors, and if it is smaller than the minimum or greater than the maximum you obtain it as a solution. I have been checking the FieldFunctions.C files in the vanilla versions and they implemented pTraits<Type>::max and pTraits<Type>::min, following your advise.

Is there any chance to get this bug fixed in FOAM-extend?

[Pablo Higuera]

I have been trying to fix the bug. in fact pTraits<Type>::max and pTraits<Type>::min exist and work in FE-3.1.

The solution (at least for me, see the final comments in the post) can be found here:

http://www.cfd-online.com/Forums/openfoam-bugs/139466-fe-3-1-parallel-issues-bcs-gmax-reduce.html#post504033

[Dominik Christ]

Fixed with commits c5a2b9d7 (master) and 02833621 (nextRelease).
@Pablo: Can you please run this with your test case and re-open this issue if there should be any trouble? Thanks!