Re: [Gfs-devel] Problem with gfs_send_objects

[Stephane P. <s.p...@gm...>]

Hi Vladimir,

> The problem is that gerris hangs in gfs_union_close () when writing
> large chunks of data on systems with fast connection
> (Infiniband+OpenFabrics and alike). It hangs on the following line when
> the data chunk size exceeds some threshold (about 12 MB)
>
> fprintf (stderr, "receiving %ld bytes from PE %d\n", length, pe);
>
> Gerris also hangs in balance events due to the same reason (mmap
> combined with MPI send/receive).

Thanks for tracking this down. This is very useful. I believe that
Pascal Ray in Paris came across similar problems but didn't pinpoint
the problem as precisely as you did.

Openfabrics seems to do some weird things with memory allocation
etc... and this seems to introduce many limitations on which unix
system calls are still valid. At some point I suspected that the
problem (hangup in parallel) could also come from openfabrics more or
less explicitly disallowing calls to fork() etc...

The problem is that I haven't seen these limitations explained clearly
anywhere. Quite often they are expressed as "this may or may not work"
in the (sparse) documentation I could find on openfabrics.

So I am not surprised that a combination of mmap() + openfabrics could
cause problems (but what is the story with the 12MB blocks? is this
documented anywhere in openfabrics?).

What is more, one would expect this type of situations to lead to
openfabrics crashing with some helpful message on what caused the
crash, not just hang...

> to keep the existing write() implementations and avoid /tmp, replace
> tmpfile() call by pipes and/or fifos in RAM.

I think the solution is to generalise the interface of (all!) the
write() methods to allow writing to either a file or a memory buffer.
This will circumvent the problem altogether.
Of course this means a lot of changes throughout the code. I will have
a go and send you the patch for testing.

cheers

Stephane