Re: [Valgrind-users] Valgrind failing for MPI program?

[Julian S. <js...@ac...>]

> I'm also getting TONS of "uninitialized value" errors with HP-MPI that I
> never got before (and some of which I have carefully tracked down, and they
> are bogus, the values are clearly initialized), but that is another
> issue....

Debugging MPI apps with Valgrind is a bit tricky, but it's certainly doable.

What you describe sounds like it could have three possible causes.  You'll
have to investigate.

1. MPI implementations like to map the network card(s) into user space,
   and push data through them, bypassing the kernel.  In this situation
   Memcheck hasn't got a clue what's going on, especially for data 
   arriving at a node, and you get flooded with false errors.

2. For similar reasons, if two processes on the same node communicate
   via a shared memory region, the results will be bad.

3. It may be that OpenMPI is providing its own implementations of
   malloc, free, new, delete, etc; that Memcheck doesn't know about,
   which will also cause chaos.

Re (1) and (2), it helps if you can get OpenMPI to make all processes
(even those on the same node) communicate via standard TCP/IP networking,
so that Memcheck can see data going into/out-of each process correctly.
Some time ago I was told by an OpenMPI developer that this can be done
by passing --mca btl tcp,self to mpirun.

Re (3), that's more difficult to ascertain.

I suggest also that asking the OpenMPI developers is worthwhile.  I've
found them in the past to be knowledgeable and helpful, and I believe they
are long-time users of Valgrind/Memcheck.

I believe you should be able to get to essentially zero false errors
with a suitable OpenMPI configuration.  I managed that in my testing with
OpenMPI a couple of years back, although I should say that was very
limited testing.

Since you are upgrading from Valgrind 2.2, once you achieve a clean run,
you might want to consider using Memcheck's MPI-checking wrapper library
for extra validation at the PMPI_* function interface level, if you haven't
already discovered that.  See 
http://www.valgrind.org/docs/manual/mc-manual.html#mc-manual.mpiwrap
for details.  I wouldn't recommend this before you get a clean run,
though; the results will be confusing.

J