From: Dmitry K. <ka...@mc...> - 2013-07-26 17:02:25
|
Dear All, It appears to me that preallocation of blocked PETSc matrices may be computed incorrectly in libMesh under certain circumstances. Consider the following 9-node mesh: 8--7--9 | | | 4--3--6 | | | 1--2--5 with two first order Lagrange variables on it. Assume that _dof_coupling is "diagonal" (i.e., the variables are not coupled to one another - a real use case in Moose, at least in moose_test). Then the "unblocked" n_nz would be (since there are 2 identical dofs per node): 4,4,6,6,9,9,6,6,4,4,6,6,6,6,4,4,4,4 The nodes, identified with the matrix blocks, have a similar connectivity, although there are half as many of them: 4,6,9,6,6,4,6,6,4,4. This blocked preallocation is transformed, however, into the following blocked version b_n_nz: 2,3,4(sic!),3,3,2,3,3,2,2, which is incorrect. This subsequently leads to extra mallocs during matrix assembly. It seems to me that correct preallocation will only result for "full coupling" -- _dof_coupling of all ones. Dmitry. |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2013-07-26 19:10:41
|
On Jul 26, 2013, at 12:02 PM, Dmitry Karpeyev <ka...@mc...> wrote: > This subsequently leads to extra mallocs during matrix assembly. It seems to me that correct preallocation will only result for "full coupling" -- _dof_coupling of all ones. Dmitry.You are absolutely correct - I will restrict the blocked implementation to be active only when all variables are coupled to each other. Thanks! -Ben |
From: Dmitry K. <ka...@mc...> - 2013-07-26 21:27:00
|
Cool. Thanks, Ben. Dmitry. On Fri, Jul 26, 2013 at 3:10 PM, Kirk, Benjamin (JSC-EG311) < ben...@na...> wrote: > On Jul 26, 2013, at 12:02 PM, Dmitry Karpeyev <ka...@mc...> wrote: > > > This subsequently leads to extra mallocs during matrix assembly. It > seems to me that correct preallocation will only result for "full coupling" > -- _dof_coupling of all ones. > > Dmitry.You are absolutely correct - I will restrict the blocked > implementation to be active only when all variables are coupled to each > other. > > Thanks! > > -Ben > > |
From: Derek G. <fri...@gm...> - 2013-07-27 06:47:10
|
Sent from my iPad On Jul 26, 2013, at 1:10 PM, "Kirk, Benjamin (JSC-EG311)" <ben...@na...> wrote: > On Jul 26, 2013, at 12:02 PM, Dmitry Karpeyev <ka...@mc...> wrote: > >> This subsequently leads to extra mallocs during matrix assembly. It seems to me that correct preallocation will only result for "full coupling" -- _dof_coupling of all ones. > > Dmitry.You are absolutely correct - I will restrict the blocked implementation to be active only when all variables are coupled to each other. Firstly: Awesome find Dmitry! Thanks! Secondly: This is a HUGE bummer. This essentially means that MOOSE-based applications will never be able to take advantage of block matrices (we never form a full matrix). Ben: is there anything that can be done here? We were really looking forward to this... Derek |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2013-07-27 11:45:57
|
On Jul 27, 2013, at 1:47 AM, "Derek Gaston" <fri...@gm...> wrote: > Ben: is there anything that can be done here? We were really looking > forward to this... So the petsc matrices assume a constant blocking factor for the entire matrix, which I think is the bug fundamental driver here. If you solved decoupled flow/temperature in 1 matrix with equal order basis functions you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think petsc will optimize for that case, and in fact you *might* be better memory wise splitting into two matrices where the flow vars could be coupled. Is there another use case with non-full dof coupling I could be overlooking? I'd be happy to try to address it... -Ben |
From: Derek G. <fri...@gm...> - 2013-07-27 14:22:05
|
Sent from my iPad On Jul 27, 2013, at 5:45 AM, "Kirk, Benjamin (JSC-EG311)" <ben...@na...> wrote: > On Jul 27, 2013, at 1:47 AM, "Derek Gaston" <fri...@gm...> wrote: > >> Ben: is there anything that can be done here? We were really looking >> forward to this... > > So the petsc matrices assume a constant blocking factor for the entire matrix, which I think is the bug fundamental driver here. If you solved decoupled flow/temperature in 1 matrix with equal order basis functions you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think petsc will optimize for that case, and in fact you *might* be better memory wise splitting into two matrices where the flow vars could be coupled. > > Is there another use case with non-full dof coupling I could be overlooking? I'd be happy to try to address it... > > -Ben > > |
From: Derek G. <fri...@gm...> - 2013-07-27 14:30:05
|
Oops accidentally hit send! Look below for the real email :-) Sent from my iPad On Jul 27, 2013, at 5:45 AM, "Kirk, Benjamin (JSC-EG311)" <ben...@na...> wrote: > So the petsc matrices assume a constant blocking factor for the entire matrix, which I think is the bug fundamental driver here. If you solved decoupled flow/temperature in 1 matrix with equal order basis functions you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think petsc will optimize for that case, and in fact you *might* be better memory wise splitting into two matrices where the flow vars could be coupled. I gotcha. There might still be a way to do this in one matrix with MatNest. Maybe you would nest two block matrices inside the matrix? > Is there another use case with non-full dof coupling I could be overlooking? I'd be happy to try to address it... Yes. The normal one for us is block-diagonal (just for preconditioning). We might have 2000 coupled variables (literally!) that are all linear lagrange.... But we use a block-diagonal preconditioning matrix. Is there something we can do about that case? Regardless of the actual coupling present in our problem we build our matrices with arbitrary amounts of off-diagonal blocks... it all depends on how important that block may or may not be for precnditioning. Derek |
From: Jed B. <jed...@mc...> - 2013-07-27 16:58:01
|
These large nearly-diagonal blocks pessimal for BAIJ. If you want to split into a few blocks that are each nearly-dense then a combination of BAIJ and Nest should be good. On Jul 27, 2013 10:30 PM, "Derek Gaston" <fri...@gm...> wrote: > Oops accidentally hit send! Look below for the real email :-) > > Sent from my iPad > > On Jul 27, 2013, at 5:45 AM, "Kirk, Benjamin (JSC-EG311)" > <ben...@na...> wrote: > > > So the petsc matrices assume a constant blocking factor for the entire > matrix, which I think is the bug fundamental driver here. If you solved > decoupled flow/temperature in 1 matrix with equal order basis functions > you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think > petsc will optimize for that case, and in fact you *might* be better memory > wise splitting into two matrices where the flow vars could be coupled. > > I gotcha. There might still be a way to do this in one matrix with > MatNest. Maybe you would nest two block matrices inside the matrix? > > > Is there another use case with non-full dof coupling I could be > overlooking? I'd be happy to try to address it... > > Yes. The normal one for us is block-diagonal (just for > preconditioning). We might have 2000 coupled variables (literally!) > that are all linear lagrange.... But we use a block-diagonal > preconditioning matrix. Is there something we can do about that case? > > Regardless of the actual coupling present in our problem we build our > matrices with arbitrary amounts of off-diagonal blocks... it all > depends on how important that block may or may not be for > precnditioning. > > Derek > |