Re: [Libmesh-users] Requirements for adding dof_constraint_rows in parallel

[Roy S. <roy...@ic...>]

On Thu, 25 Jan 2018, David Knezevic wrote:

> Thanks for that info. I'm only using ReplicatedMesh for now. Regarding this:
>
>       IIRC: on ReplicatedMesh we try to avoid communication as much as possible, so you have to add user constraints identically on
>       every processor.  There might be ways to get around that (add local constraints only, then re-add after any repartitioning?) but
>       I definitely wouldn't suggest trying to depend on them.
> 
> hmm, the reason I asked about this was because I had a case that hit
> a "cyclic loop" assertion in parallel when I added the constraints
> identically on every processor,
>
> and then when I changed it so that I only added the constraint on
> the processor that owned the constrained dof, it worked fine. I
> assumed that I must have been violating an assumption by adding the
> constraints identically on all processors, but your comment would
> indicate that isn't the case...

Looking back at dof_map_constraints.C, it looks like my comment was
wrong - I did parallelize the constraint calculations, even on
ReplicatedMesh.  We're looping only over local elements when we
compute hanging node and Dirichlet constraints.  That means we *must*
communicate (e.g. to handle cases where a local constraint only gets
computed on a ghost element) later.

> I'd like to understand this better. Can you point me to the relevant
> code in the library? I've been looking through
> allgather_recursive_constraints and similar, and I figured they send
> and receive constraint rows as needed, which is why I assumed we
> didn't need to add constraint rows identically on all processors.

The first section of allgather_recursive_constraints() is what's
relevant here - it does the initial push of constraints that we've
calculated to their owners.

And based on that code, it looks like the rule is "we don't need to
add constraint rows on all processors, but if we *do* then we need to
add them identically".

It might be worth adding some error checking there to make sure that
happens.  Right now if a processor has a constrained row, and gets
pushed a constraint for that row, it assumes the remote constraint is
equivalent to the local and it ignores it.  We should probably
libmesh_assert equivalency (within TOLERANCE*TOLERANCE ought to be
safe; these are nondimensional numbers) instead.
---
Roy