These large nearly-diagonal blocks pessimal for BAIJ. If you want to split into a few blocks that are each nearly-dense then a combination of BAIJ and Nest should be good.

On Jul 27, 2013 10:30 PM, "Derek Gaston" <friedmud@gmail.com> wrote:

Oops accidentally hit send! Look below for the real email :-)

Sent from my iPad

On Jul 27, 2013, at 5:45 AM, "Kirk, Benjamin (JSC-EG311)"

<benjamin.kirk@nasa.gov> wrote:

> So the petsc matrices assume a constant blocking factor for the entire matrix, which I think is the bug fundamental driver here. If you solved decoupled flow/temperature in 1 matrix with equal order basis functions you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think petsc will optimize for that case, and in fact you *might* be better memory wise splitting into two matrices where the flow vars could be coupled.

I gotcha. There might still be a way to do this in one matrix with

MatNest. Maybe you would nest two block matrices inside the matrix?

> Is there another use case with non-full dof coupling I could be overlooking? I'd be happy to try to address it...

Yes. The normal one for us is block-diagonal (just for

preconditioning). We might have 2000 coupled variables (literally!)

that are all linear lagrange.... But we use a block-diagonal

preconditioning matrix. Is there something we can do about that case?

Regardless of the actual coupling present in our problem we build our

matrices with arbitrary amounts of off-diagonal blocks... it all

depends on how important that block may or may not be for

precnditioning.

Derek