Re: [Libmesh-users] Requirements for adding dof_constraint_rows in parallel

[David K. <dav...@ak...>]

On Fri, Jan 26, 2018 at 11:06 AM, Roy Stogner <roy...@ic...>
wrote:

>
> On Thu, 25 Jan 2018, David Knezevic wrote:
>
> Thanks for that info. I'm only using ReplicatedMesh for now. Regarding
>> this:
>>
>>       IIRC: on ReplicatedMesh we try to avoid communication as much
>> as possible, so you have to add user constraints identically on
>>       every processor.  There might be ways to get around that (add
>> local constraints only, then re-add after any repartitioning?) but
>>       I definitely wouldn't suggest trying to depend on them.
>>
>> hmm, the reason I asked about this was because I had a case that hit
>> a "cyclic loop" assertion in parallel when I added the constraints
>> identically on every processor,
>>
>> and then when I changed it so that I only added the constraint on
>> the processor that owned the constrained dof, it worked fine. I
>> assumed that I must have been violating an assumption by adding the
>> constraints identically on all processors, but your comment would
>> indicate that isn't the case...
>>
>
> Looking back at dof_map_constraints.C, it looks like my comment was
> wrong - I did parallelize the constraint calculations, even on
> ReplicatedMesh.  We're looping only over local elements when we
> compute hanging node and Dirichlet constraints.  That means we *must*
> communicate (e.g. to handle cases where a local constraint only gets
> computed on a ghost element) later.



OK, thanks for confirming that. Yes, I see that we only loop over local
elements in DofMap::create_dof_constraints.



> I'd like to understand this better. Can you point me to the relevant
>> code in the library? I've been looking through
>> allgather_recursive_constraints and similar, and I figured they send
>> and receive constraint rows as needed, which is why I assumed we
>> didn't need to add constraint rows identically on all processors.
>>
>
> The first section of allgather_recursive_constraints() is what's
> relevant here - it does the initial push of constraints that we've
> calculated to their owners.
>
> And based on that code, it looks like the rule is "we don't need to
> add constraint rows on all processors, but if we *do* then we need to
> add them identically".
>

OK, that makes sense. Yeah, I see where we have the comment "If we don't
already have a constraint for this dof, add the one we were sent"



> It might be worth adding some error checking there to make sure that
> happens.  Right now if a processor has a constrained row, and gets
> pushed a constraint for that row, it assumes the remote constraint is
> equivalent to the local and it ignores it.  We should probably
> libmesh_assert equivalency (within TOLERANCE*TOLERANCE ought to be
> safe; these are nondimensional numbers) instead.
>

Agreed, that seems like a good idea. I guess that check could be added near
the comment that I pasted above.


Thanks for your help! That clarifies things a lot!

Regards,
David