Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.
Close
From: Dmitry Karpeyev <karpeev@mc...>  20130726 17:02:25
Attachments:
Message as HTML

Dear All, It appears to me that preallocation of blocked PETSc matrices may be computed incorrectly in libMesh under certain circumstances. Consider the following 9node mesh: 879    436    125 with two first order Lagrange variables on it. Assume that _dof_coupling is "diagonal" (i.e., the variables are not coupled to one another  a real use case in Moose, at least in moose_test). Then the "unblocked" n_nz would be (since there are 2 identical dofs per node): 4,4,6,6,9,9,6,6,4,4,6,6,6,6,4,4,4,4 The nodes, identified with the matrix blocks, have a similar connectivity, although there are half as many of them: 4,6,9,6,6,4,6,6,4,4. This blocked preallocation is transformed, however, into the following blocked version b_n_nz: 2,3,4(sic!),3,3,2,3,3,2,2, which is incorrect. This subsequently leads to extra mallocs during matrix assembly. It seems to me that correct preallocation will only result for "full coupling"  _dof_coupling of all ones. Dmitry. 
From: Kirk, Benjamin (JSCEG311) <benjamin.kirk@na...>  20130726 19:10:41

On Jul 26, 2013, at 12:02 PM, Dmitry Karpeyev <karpeev@...> wrote: > This subsequently leads to extra mallocs during matrix assembly. It seems to me that correct preallocation will only result for "full coupling"  _dof_coupling of all ones. Dmitry.You are absolutely correct  I will restrict the blocked implementation to be active only when all variables are coupled to each other. Thanks! Ben 
From: Dmitry Karpeyev <karpeev@mc...>  20130726 21:27:00
Attachments:
Message as HTML

Cool. Thanks, Ben. Dmitry. On Fri, Jul 26, 2013 at 3:10 PM, Kirk, Benjamin (JSCEG311) < benjamin.kirk@...> wrote: > On Jul 26, 2013, at 12:02 PM, Dmitry Karpeyev <karpeev@...> wrote: > > > This subsequently leads to extra mallocs during matrix assembly. It > seems to me that correct preallocation will only result for "full coupling" >  _dof_coupling of all ones. > > Dmitry.You are absolutely correct  I will restrict the blocked > implementation to be active only when all variables are coupled to each > other. > > Thanks! > > Ben > > 
From: Derek Gaston <friedmud@gm...>  20130727 06:47:10

Sent from my iPad On Jul 26, 2013, at 1:10 PM, "Kirk, Benjamin (JSCEG311)" <benjamin.kirk@...> wrote: > On Jul 26, 2013, at 12:02 PM, Dmitry Karpeyev <karpeev@...> wrote: > >> This subsequently leads to extra mallocs during matrix assembly. It seems to me that correct preallocation will only result for "full coupling"  _dof_coupling of all ones. > > Dmitry.You are absolutely correct  I will restrict the blocked implementation to be active only when all variables are coupled to each other. Firstly: Awesome find Dmitry! Thanks! Secondly: This is a HUGE bummer. This essentially means that MOOSEbased applications will never be able to take advantage of block matrices (we never form a full matrix). Ben: is there anything that can be done here? We were really looking forward to this... Derek 
From: Kirk, Benjamin (JSCEG311) <benjamin.kirk@na...>  20130727 11:45:57

On Jul 27, 2013, at 1:47 AM, "Derek Gaston" <friedmud@...> wrote: > Ben: is there anything that can be done here? We were really looking > forward to this... So the petsc matrices assume a constant blocking factor for the entire matrix, which I think is the bug fundamental driver here. If you solved decoupled flow/temperature in 1 matrix with equal order basis functions you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think petsc will optimize for that case, and in fact you *might* be better memory wise splitting into two matrices where the flow vars could be coupled. Is there another use case with nonfull dof coupling I could be overlooking? I'd be happy to try to address it... Ben 
From: Derek Gaston <friedmud@gm...>  20130727 14:22:05

Sent from my iPad On Jul 27, 2013, at 5:45 AM, "Kirk, Benjamin (JSCEG311)" <benjamin.kirk@...> wrote: > On Jul 27, 2013, at 1:47 AM, "Derek Gaston" <friedmud@...> wrote: > >> Ben: is there anything that can be done here? We were really looking >> forward to this... > > So the petsc matrices assume a constant blocking factor for the entire matrix, which I think is the bug fundamental driver here. If you solved decoupled flow/temperature in 1 matrix with equal order basis functions you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think petsc will optimize for that case, and in fact you *might* be better memory wise splitting into two matrices where the flow vars could be coupled. > > Is there another use case with nonfull dof coupling I could be overlooking? I'd be happy to try to address it... > > Ben > > 
From: Derek Gaston <friedmud@gm...>  20130727 14:30:05

Oops accidentally hit send! Look below for the real email :) Sent from my iPad On Jul 27, 2013, at 5:45 AM, "Kirk, Benjamin (JSCEG311)" <benjamin.kirk@...> wrote: > So the petsc matrices assume a constant blocking factor for the entire matrix, which I think is the bug fundamental driver here. If you solved decoupled flow/temperature in 1 matrix with equal order basis functions you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think petsc will optimize for that case, and in fact you *might* be better memory wise splitting into two matrices where the flow vars could be coupled. I gotcha. There might still be a way to do this in one matrix with MatNest. Maybe you would nest two block matrices inside the matrix? > Is there another use case with nonfull dof coupling I could be overlooking? I'd be happy to try to address it... Yes. The normal one for us is blockdiagonal (just for preconditioning). We might have 2000 coupled variables (literally!) that are all linear lagrange.... But we use a blockdiagonal preconditioning matrix. Is there something we can do about that case? Regardless of the actual coupling present in our problem we build our matrices with arbitrary amounts of offdiagonal blocks... it all depends on how important that block may or may not be for precnditioning. Derek 
From: Jed Brown <jedbrown@mc...>  20130727 16:58:01
Attachments:
Message as HTML

These large nearlydiagonal blocks pessimal for BAIJ. If you want to split into a few blocks that are each nearlydense then a combination of BAIJ and Nest should be good. On Jul 27, 2013 10:30 PM, "Derek Gaston" <friedmud@...> wrote: > Oops accidentally hit send! Look below for the real email :) > > Sent from my iPad > > On Jul 27, 2013, at 5:45 AM, "Kirk, Benjamin (JSCEG311)" > <benjamin.kirk@...> wrote: > > > So the petsc matrices assume a constant blocking factor for the entire > matrix, which I think is the bug fundamental driver here. If you solved > decoupled flow/temperature in 1 matrix with equal order basis functions > you'd have 4 fully coupled variables in 3D, with 1 tagalog. I don't think > petsc will optimize for that case, and in fact you *might* be better memory > wise splitting into two matrices where the flow vars could be coupled. > > I gotcha. There might still be a way to do this in one matrix with > MatNest. Maybe you would nest two block matrices inside the matrix? > > > Is there another use case with nonfull dof coupling I could be > overlooking? I'd be happy to try to address it... > > Yes. The normal one for us is blockdiagonal (just for > preconditioning). We might have 2000 coupled variables (literally!) > that are all linear lagrange.... But we use a blockdiagonal > preconditioning matrix. Is there something we can do about that case? > > Regardless of the actual coupling present in our problem we build our > matrices with arbitrary amounts of offdiagonal blocks... it all > depends on how important that block may or may not be for > precnditioning. > > Derek > 