Ok - MeshCommunication::assign_global_indices() is beyond my comprehension... BUT there are a couple of "#if 0" error checks in there that if un-#if'd they do trigger these things with my test case:
Error: nodes with duplicate Hilbert keys!
node 7, (x,y,z)=( 0.5, 0.5, 0) has HilbertIndices 3758096384_0_0
node 9, (x,y,z)=( 0.5, 0.5, 0) has HilbertIndices 3758096384_0_0
So I do suspect that to be the issue...
So what to do? We basically shouldn't be relying on Hilbert indices for this part of libMesh.
Is it time for a new IO type for EquationSystems? Should I do an EquationSystemsCheckpointIO class like I did for the mesh? If you don't care about m->n restart then it should be unbelievably simple (each processor just writes it's local data) and it should scale better in parallel than the current stuff (which ultimately serializes through processor 0).