Re: [Libmesh-users] Problem when running NOX in parallel

[Derek G. <fri...@gm...>]

Quick aside before I get into my real reply: Can we get Norbert added  
to the libmesh-devel list and take this discussion there?  No reason  
to be hitting all of our users with this stuff.

Now my reply:

On Oct 16, 2008, at 9:16 AM, Norbert Stoop wrote:

> What exactly is the missing functionality there? If I understand
> correctly, libmesh is already using the FECrsMatrix, so summation of  
> the
> off-processor contributions should be taken care of already, or am I
> missing something?

The issue is that NOX _doesn't_ use FECrsMatrix... and instead just  
uses a CrsMatrix.  The trouble with this is that when they give you  
back a pointer to the jacobian to fill... it comes through as a  
CrsMatrix which has no parallel summation capability.  With FEVector  
this was particularly bad because there was no way to create an  
FEVector from an EpetraVector.  This is why we now _don't_ use  
FEVector... and instead go to great lengths to work directly with an  
EpetraVector and do the off-processor summations external to the  
object itself.

Now... from what I've seen of FECrsMatrix it _does_ have the "View"  
capability.... I was hoping to exploit that to create a "View" of the  
data in the CrsMatrix that NOX passes us back.... and then still use  
FECrsMatrix's off-processor summing capabilities.  But I don't know  
yet whether or not this will work.  If it doesn't... then we're in the  
same place we were with FEVector... we'll do all of the off-processor  
summation work ourselves and just use a CrsMatrix instead of an  
FECrsMatrix.

Hope that makes sense.

> Ok, I'll do that as soon as I have made some more tests.

As long as you're making tests... can you turn that LaplaceYoung  
example into a _real_example (just clean it up a bit).  We really need  
to get that thing committed so that people have something to look at  
for how to use the NonlinearSolver classes.

> Sorry, it seems I was somewhere else with my thoughts when I wrote  
> this
> - it's quite obvious, as you point out, that the problem must be in  
> the
> summation. I was only wondering why the linear problems do not suffer
> from the same bug. Now, it seems like the only difference between ex4,
> which works fine, and the nonlinear one is that ex4 makes no calls to
> inputNonlocalValue(), while ex19 does. If I understand correctly,
> GlobalAssemble should then update the overlap, right?

That's right.

> Ah, ok, so the resolution is ridiculously simple: Parallel summation
> seems to be just fine (I did not check all of the code, mainly the
> nonlocal values stuff), it's just that the residual vector is never
> closed in the computeF callback! Doing so (like in the petsc version)
> resolves everything! Had I only read through the
> petsc_nonlinear_solver.C more carefully...

Great!  So maybe I did forget to commit that fix... doh.  Sorry about  
that.  If you could include that in your patches I would appreciate it!

Thanks a bunch for working on this!

Derek