Ok - MeshCommunication::assign_global_indices() is beyond my comprehension... BUT there are a couple of "#if 0" error checks in there that if un-#if'd they do trigger these things with my test case:

Error: nodes with duplicate Hilbert keys!
node 7, (x,y,z)=(     0.5,      0.5,        0) has HilbertIndices 3758096384_0_0
node 9, (x,y,z)=(     0.5,      0.5,        0) has HilbertIndices 3758096384_0_0

So I do suspect that to be the issue...

So what to do?  We basically shouldn't be relying on Hilbert indices for this part of libMesh.

Is it time for a new IO type for EquationSystems?  Should I do an EquationSystemsCheckpointIO class like I did for the mesh?  If you don't care about m->n restart then it should be unbelievably simple (each processor just writes it's local data) and it should scale better in parallel than the current stuff (which ultimately serializes through processor 0).



On Fri, Jan 3, 2014 at 8:17 AM, Roy Stogner <roystgnr@ices.utexas.edu> wrote:

On Thu, 2 Jan 2014, Derek Gaston wrote:

One of my users brought an issue to my attention: libMesh is not properly handling multiple nodes occupying the same
position when writing out xda/xdr files for the solution vector.
I am attaching a mesh and main.C that show the issue.  If you look at the resulting eq_sys.xda file you will see that it
has two zeros at the end of the solution vector.  Those two zeros should be "4.0".

The mesh has 2 pairs of nodes that are "duplicated".  They are at (0, 0.5, 0) and (0.5, 0.5, 0).  What I mean by
duplicated is that there are two nodes in the same physical position (so there is a "slit" in the mesh).  Think of it
like a "crack" in the mesh...

Solving on a mesh like this works just fine... but something about the XDA/R output routines doesn't like it (I suspect
it's the Hilbert space reorder... but that's just a guess).  Note that the Exodus output looks perfectly fine - as does
the print of the solution vector.  It really is an issue just with XDA/R

I would really appreciate help on this one!

IIRC there was a problem with find_neighbors() that was causing meshes
with slits to fail at AMR, and so I've never really investigated the
compatibility of the rest of the library with those cases.  I'd
definitely also suspect the Hilbert reordering as the culprit here.