#310 Crash with interDyMFoam + dynamicRefineFvMesh + ~+48 procs

[Roberto Ribeiro]

Hi,

I'm facing a crash with a practically out-of-box run with foam-extend-3.1 (tested also with 3.2 and 4.0).

Basically, I'm grabbing the damBreakWithObstacle tutorial, adjusting it to run with 64procs and I'm getting the following error at an arbitrary time-step:

Courant Number mean: 0.00153715 max: 0.111756 velocity magnitude: 1.28822
Time = 0.017

Selected 5 cells for refinement out of 102711.

    From function void polyMesh::initMesh()
    in file meshes/polyMesh/polyMeshInitMesh.C at line 81
    Truncating neighbour list at 22329 for backward compatibility
Refined from 102711 to 102746 cells.
Selected 0 split points out of a possible 8633.
time step continuity errors : sum local = 2.9183e-06, global = -5.04178e-11, cumulative = -2.40664e-05
GAMGPCG:  Solving for pcorr, Initial residual = 1, Final residual = 2.78744e-05, No Iterations 4
time step continuity errors : sum local = 1.02223e-09, global = 3.25557e-14, cumulative = -2.40664e-05
Courant Number mean: 0.00153664 max: 0.108621 velocity magnitude: 1.05436
MULES: Solving for alpha1
Liquid phase volume fraction = 0.0848214  Min(alpha1) = -7.32253e-22  Max(alpha1) = 1
MULES: Solving for alpha1
Liquid phase volume fraction = 0.0848214  Min(alpha1) = -1.49078e-22  Max(alpha1) = 1
MULES: Solving for alpha1
Liquid phase volume fraction = 0.0848214  Min(alpha1) = -1.21907e-22  Max(alpha1) = 1
smoothSolver:  Solving for Ux, Initial residual = 0.0297867, Final residual = 2.39419e-08, No Iterations 3
smoothSolver:  Solving for Uy, Initial residual = 0.0290037, Final residual = 2.9586e-08, No Iterations 3
smoothSolver:  Solving for Uz, Initial residual = 0.0495207, Final residual = 3.42538e-08, No Iterations 3
GAMG:  Solving for pd, Initial residual = 0.00403362, Final residual = 0.000170153, No Iterations 1
time step continuity errors : sum local = 7.24378e-05, global = 3.60611e-09, cumulative = -2.40628e-05
GAMG:  Solving for pd, Initial residual = 0.000210132, Final residual = 6.90479e-06, No Iterations 3
time step continuity errors : sum local = 2.92656e-06, global = 5.70459e-08, cumulative = -2.40058e-05
GAMG:  Solving for pd, Initial residual = 2.78919e-05, Final residual = 1.33396e-06, No Iterations 3
time step continuity errors : sum local = 5.65676e-07, global = 1.19196e-08, cumulative = -2.39939e-05
GAMGPCG:  Solving for pd, Initial residual = 5.38728e-06, Final residual = 9.72962e-09, No Iterations 3
time step continuity errors : sum local = 4.12615e-09, global = -5.75693e-11, cumulative = -2.39939e-05
ExecutionTime = 7.64 s  ClockTime = 12 s

Courant Number mean: 0.00162271 max: 0.111354 velocity magnitude: 1.27342
Time = 0.018

Selected 4 cells for refinement out of 102746.

    From function void polyMesh::initMesh()
    in file meshes/polyMesh/polyMeshInitMesh.C at line 81
    Truncating neighbour list at 22329 for backward compatibility
Refined from [48]
[48]
[48] --> FOAM FATAL IO ERROR:
[48] incorrect first token, expected <int> or '(', found on line 0 the word 'x'
[48]
[48] file: IOstream at line 0.
[48]
[48]     From function operator>>(Istream&, List<T>&)
[48]     in file /work/01502/rri/work/openfoam/extend-3.1/builds/master-sandy/foam-extend-3.1/src/foam/lnInclude/ListIO.C at line 149.
[48]
FOAM parallel run exiting
[48]
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 48

The error is however presented in different ways, like nan in mutliple residuals or floating point exceptions. The above one is the most common. It also seems to present with high core counts like 48+

I'm running this in a cluster with IntelMPI and compiled OF with ICC. I also tried with MVAPICH2 and GCC. The run used 4 identical nodes. The mesh size is default (but also tried with higher cell count), I disabled the adpativeTimeStep for simplicity, and these were basically my only changes.

Took deliverity to declare this as bug, but any help would be appreciated with workaround or explanation
Thanks