From: Roy Stogner <roystgnr@ic...>  20090929 20:49:13

On Tue, 29 Sep 2009, Derek Gaston wrote: > Is there something specific I can do to recreate the bug to help test? Verifying "where does the bug appear" is done, and the answer is "everywhere"; after reverting some application changes at home I can reproduce it there too. I think I've identified the bug. It appears to be a consequence of using firstorder finite elements with secondorder geometric elements, and it's only a serious problem when using asymmetric_constraint_rows=false, which combination may be why most people never noticed it earlier. Vikram just happens to be using an older version of some application framework code of mine which defaults to bumping up geometric element order but which he'd dropped to bilinears for the approximation order. When I switch the geometric elements down to QUAD4 instead of QUAD9, the bug goes away. Unlike our earlier problem that the ghosted vectors caught, this one isn't due to failure to communicate dependencies; it's a failure to calculate results: ABC       D E F       GHIJK    L M N O P    QRSTU Element GKCA is on proc 1, elements QSIG and SUKI are on proc 0. Because node I is on element AGKC, even though its DoF isn't one of a bilinear elements local DoFs, the DoF still gets assigned to proc 0. When DofMap::enforce_constraints_exactly() tries to apply constraint equations, proc 1 gets the appropriate constraint matrix from both QSIG and SUKI, but in each case it recognizes that it doesn't own the DoF(s) on node I and it doesn't change them. Element GKCA is the coarse element, so its constraint matrix doesn't involve node I, and so proc 0 doesn't change node I's value either. But when using asymmetric_constraint_rows=false (whether directly or using FEMSystem, you're also expecting those constrained DoFs to be fixed up by enforce_constraints_exactly() later. That didn't happen, which led our solutions on that node to be off by a fraction of a percent and our error indicator values to be wildly off. I'm still pondering the best solution. I think it will fix the problem (and be much more efficient) if we have enforce_constraints_exactly() loop over DofConstraints entries rather than over elements. I'll let everyone know once that's committed. In the meantime, the workaround is to avoid the combination of parallel AMR/C, nodal FE types, second order Elem types, and asymmetric_constraint_rows=false (directly or via FEMSystem). I'll post that to libmeshusers too.  Roy 