These large nearly-diagonal blocks pessimal for BAIJ. If you want to split into a few blocks that are each nearly-dense then a combination of BAIJ and Nest should be good.

On Jul 27, 2013 10:30 PM, "Derek Gaston" <friedmud@gmail.com> wrote:
Oops accidentally hit send!  Look below for the real email :-)

Sent from my iPad

On Jul 27, 2013, at 5:45 AM, "Kirk, Benjamin (JSC-EG311)"
<benjamin.kirk@nasa.gov> wrote:

> So the petsc matrices assume a constant blocking factor for the entire matrix, which I think is the bug fundamental driver here. If you solved decoupled flow/temperature in 1 matrix with equal order basis functions you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think petsc will optimize for that case, and in fact you *might* be better memory wise splitting into two matrices where the flow vars could be coupled.

I gotcha.  There might still be a way to do this in one matrix with
MatNest.  Maybe you would nest two block matrices inside the matrix?

> Is there another use case with non-full dof coupling I could be overlooking?  I'd be happy to try to address it...

Yes. The normal one for us is block-diagonal (just for
preconditioning).  We might have 2000 coupled variables (literally!)
that are all linear lagrange.... But we use a block-diagonal
preconditioning matrix.  Is there something we can do about that case?

Regardless of the actual coupling present in our problem we build our
matrices with arbitrary amounts of off-diagonal blocks... it all
depends on how important that block may or may not be for
precnditioning.

Derek