libmesh-users

[Libmesh-users] BoomerAMG/hypre with CG for elasticity

[Dana C. <dan...@ak...>]

Hello,

I'm trying to solve a linear elasticity problem based on a tetraheadral
mesh with over a million degrees of freedom.

I've been using the LU solver in MUMPS but I'm now forced to switch to an
iterative solver because of the memory requirements of that solver.

CG and BoomerAMG in hypre seem to be a good combination, but I've not been
able to achieve convergence. The residual never goes below 1 even after
several thousand iterations and shows high oscillations.

The parameters I'm using are
-ksp_type cg
-ksp_monitor
-ksp_converged_reason
-pc_type hypre
-pc_hypre_type boomeramg
-pc_hypre_boomeramg_strong_threshold 0.7
-log_summary
-eps_monitor
-st_type sinvert

Does anyone have recommendations / prior experience on successfully using
BoomerAMG in libMesh?

Thanks,
Dana

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[John P. <jwp...@gm...>]

On Fri, Jan 31, 2014 at 10:04 AM, Dana Christen
<dan...@ak...>wrote:

> Hello,
>
> I'm trying to solve a linear elasticity problem based on a tetraheadral
> mesh with over a million degrees of freedom.
>
> I've been using the LU solver in MUMPS but I'm now forced to switch to an
> iterative solver because of the memory requirements of that solver.
>
> CG and BoomerAMG in hypre seem to be a good combination, but I've not been
> able to achieve convergence. The residual never goes below 1 even after
> several thousand iterations and shows high oscillations.
>
> The parameters I'm using are
> -ksp_type cg
> -ksp_monitor
> -ksp_converged_reason
> -pc_type hypre
> -pc_hypre_type boomeramg
> -pc_hypre_boomeramg_strong_threshold 0.7
> -log_summary
>

First thing I would think of is that the problem isn't actually symmetric
even though logically it should be?

Does it work any better with -ksp_type bcgs?



> -eps_monitor
> -st_type sinvert
>

Is this an eigenvalue problem too?



-- 
John

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[David K. <dkn...@se...>]

On 01/31/2014 12:36 PM, John Peterson wrote:
>> Hello,
>>
>> I'm trying to solve a linear elasticity problem based on a tetraheadral
>> mesh with over a million degrees of freedom.
>>
>> I've been using the LU solver in MUMPS but I'm now forced to switch to an
>> iterative solver because of the memory requirements of that solver.
>>
>> CG and BoomerAMG in hypre seem to be a good combination, but I've not been
>> able to achieve convergence. The residual never goes below 1 even after
>> several thousand iterations and shows high oscillations.
>>
>> The parameters I'm using are
>> -ksp_type cg
>> -ksp_monitor
>> -ksp_converged_reason
>> -pc_type hypre
>> -pc_hypre_type boomeramg
>> -pc_hypre_boomeramg_strong_threshold 0.7
>> -log_summary
>>
> First thing I would think of is that the problem isn't actually symmetric
> even though logically it should be?
>
> Does it work any better with -ksp_type bcgs?
>
>
>
>> -eps_monitor
>> -st_type sinvert
>>
> Is this an eigenvalue problem too?

Dana is working with me on this. It's not an eigenproblem, so 
"-eps_monitor -st_type sinvert" isn't meant to be there (but it those 
options would've just been ignored I believe).

Regarding symmetry: That's a good thought. The PDE is linear elasticity, 
hence it's symmetric. But we enforce boundary conditions using 
DirchletBoundary, so that'll ruin symmetry on the Dirichlet rows/cols. 
Not sure if that's the problem here or not, but Dana will try "-ksp_type 
bcgs" to see if that helps convergence.

Thanks,
David

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[Jens L. E. <jle...@gm...>]

Hi,

I used these arguments for 3D elasticity and they seemed to work well 
also in combination with DirichletBoundary (why does that break symmetry?)

-ksp_type cg
-pc_type fieldsplit
-pc_fieldsplit_block_size 3
-fieldsplit_pc_type hypre
-fieldsplit_pc_hypre_type boomeramg
-fieldsplit_pc_hypre_boomeramg_strong_threshold 0.7
-pc_fieldsplit_0 0,1,2
-ksp_atol 1e-11
-ksp_rtol 1e-11
-pc_fieldsplit_type symmetric_multiplicative

It is key to use the fieldsplit preconditioner in order to be able to 
exploit smoothness within each field component. If it's 2D elasticity I 
believe you'd have to modify to -pc_fieldsplit_block_size 2 and 
-pc_fieldsplit_0 0,1.

I was able to solve a problem with tens of millions dofs in 90 CG 
iterations using these parameters.

Best,
Jens

On 01/31/2014 11:11 PM, David Knezevic wrote:
> On 01/31/2014 12:36 PM, John Peterson wrote:
>>> Hello,
>>>
>>> I'm trying to solve a linear elasticity problem based on a tetraheadral
>>> mesh with over a million degrees of freedom.
>>>
>>> I've been using the LU solver in MUMPS but I'm now forced to switch to an
>>> iterative solver because of the memory requirements of that solver.
>>>
>>> CG and BoomerAMG in hypre seem to be a good combination, but I've not been
>>> able to achieve convergence. The residual never goes below 1 even after
>>> several thousand iterations and shows high oscillations.
>>>
>>> The parameters I'm using are
>>> -ksp_type cg
>>> -ksp_monitor
>>> -ksp_converged_reason
>>> -pc_type hypre
>>> -pc_hypre_type boomeramg
>>> -pc_hypre_boomeramg_strong_threshold 0.7
>>> -log_summary
>>>
>> First thing I would think of is that the problem isn't actually symmetric
>> even though logically it should be?
>>
>> Does it work any better with -ksp_type bcgs?
>>
>>
>>
>>> -eps_monitor
>>> -st_type sinvert
>>>
>> Is this an eigenvalue problem too?
> Dana is working with me on this. It's not an eigenproblem, so
> "-eps_monitor -st_type sinvert" isn't meant to be there (but it those
> options would've just been ignored I believe).
>
> Regarding symmetry: That's a good thought. The PDE is linear elasticity,
> hence it's symmetric. But we enforce boundary conditions using
> DirchletBoundary, so that'll ruin symmetry on the Dirichlet rows/cols.
> Not sure if that's the problem here or not, but Dana will try "-ksp_type
> bcgs" to see if that helps convergence.
>
> Thanks,
> David
>
>
>
>
> ------------------------------------------------------------------------------
> WatchGuard Dimension instantly turns raw network data into actionable
> security intelligence. It gives you real-time visual feedback on key
> security issues and trends.  Skip the complicated setup - simply import
> a virtual appliance and go from zero to informed in seconds.
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> _______________________________________________
> Libmesh-users mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[David K. <dkn...@se...>]

Thanks Jens, we'll try this next week.



On 02/01/2014 04:06 AM, Jens Lohne Eftang wrote:
> Hi,
>
> I used these arguments for 3D elasticity and they seemed to work well
> also in combination with DirichletBoundary (why does that break symmetry?)
>
> -ksp_type cg
> -pc_type fieldsplit
> -pc_fieldsplit_block_size 3
> -fieldsplit_pc_type hypre
> -fieldsplit_pc_hypre_type boomeramg
> -fieldsplit_pc_hypre_boomeramg_strong_threshold 0.7
> -pc_fieldsplit_0 0,1,2
> -ksp_atol 1e-11
> -ksp_rtol 1e-11
> -pc_fieldsplit_type symmetric_multiplicative
>
> It is key to use the fieldsplit preconditioner in order to be able to
> exploit smoothness within each field component. If it's 2D elasticity I
> believe you'd have to modify to -pc_fieldsplit_block_size 2 and
> -pc_fieldsplit_0 0,1.
>
> I was able to solve a problem with tens of millions dofs in 90 CG
> iterations using these parameters.
>
> Best,
> Jens
>
> On 01/31/2014 11:11 PM, David Knezevic wrote:
>> On 01/31/2014 12:36 PM, John Peterson wrote:
>>>> Hello,
>>>>
>>>> I'm trying to solve a linear elasticity problem based on a tetraheadral
>>>> mesh with over a million degrees of freedom.
>>>>
>>>> I've been using the LU solver in MUMPS but I'm now forced to switch to an
>>>> iterative solver because of the memory requirements of that solver.
>>>>
>>>> CG and BoomerAMG in hypre seem to be a good combination, but I've not been
>>>> able to achieve convergence. The residual never goes below 1 even after
>>>> several thousand iterations and shows high oscillations.
>>>>
>>>> The parameters I'm using are
>>>> -ksp_type cg
>>>> -ksp_monitor
>>>> -ksp_converged_reason
>>>> -pc_type hypre
>>>> -pc_hypre_type boomeramg
>>>> -pc_hypre_boomeramg_strong_threshold 0.7
>>>> -log_summary
>>>>
>>> First thing I would think of is that the problem isn't actually symmetric
>>> even though logically it should be?
>>>
>>> Does it work any better with -ksp_type bcgs?
>>>
>>>
>>>
>>>> -eps_monitor
>>>> -st_type sinvert
>>>>
>>> Is this an eigenvalue problem too?
>> Dana is working with me on this. It's not an eigenproblem, so
>> "-eps_monitor -st_type sinvert" isn't meant to be there (but it those
>> options would've just been ignored I believe).
>>
>> Regarding symmetry: That's a good thought. The PDE is linear elasticity,
>> hence it's symmetric. But we enforce boundary conditions using
>> DirchletBoundary, so that'll ruin symmetry on the Dirichlet rows/cols.
>> Not sure if that's the problem here or not, but Dana will try "-ksp_type
>> bcgs" to see if that helps convergence.
>>
>> Thanks,
>> David
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> WatchGuard Dimension instantly turns raw network data into actionable
>> security intelligence. It gives you real-time visual feedback on key
>> security issues and trends.  Skip the complicated setup - simply import
>> a virtual appliance and go from zero to informed in seconds.
>> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Libmesh-users mailing list
>> Lib...@li...
>> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>
> ------------------------------------------------------------------------------
> WatchGuard Dimension instantly turns raw network data into actionable
> security intelligence. It gives you real-time visual feedback on key
> security issues and trends.  Skip the complicated setup - simply import
> a virtual appliance and go from zero to informed in seconds.
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> _______________________________________________
> Libmesh-users mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[Dana C. <dan...@ak...>]

Hi,

Thanks for your assistance.

Switching to BiCGStab (-ksp_type bcgs) doesn't improve convergence and
shows the same symptoms (residual with frequent oscillations).
Using fieldsplit as Jens suggests doesn't help either; residual doesn't go
below 1e-3 even after 1000 iterations.
Using GMRES makes the oscillations disappear, but the residual doesn't
converge either.

Cheers,
Dana


On Sat, Feb 1, 2014 at 4:28 PM, David Knezevic
<dkn...@se...>wrote:

> Thanks Jens, we'll try this next week.
>
>
>
> On 02/01/2014 04:06 AM, Jens Lohne Eftang wrote:
> > Hi,
> >
> > I used these arguments for 3D elasticity and they seemed to work well
> > also in combination with DirichletBoundary (why does that break
> symmetry?)
> >
> > -ksp_type cg
> > -pc_type fieldsplit
> > -pc_fieldsplit_block_size 3
> > -fieldsplit_pc_type hypre
> > -fieldsplit_pc_hypre_type boomeramg
> > -fieldsplit_pc_hypre_boomeramg_strong_threshold 0.7
> > -pc_fieldsplit_0 0,1,2
> > -ksp_atol 1e-11
> > -ksp_rtol 1e-11
> > -pc_fieldsplit_type symmetric_multiplicative
> >
> > It is key to use the fieldsplit preconditioner in order to be able to
> > exploit smoothness within each field component. If it's 2D elasticity I
> > believe you'd have to modify to -pc_fieldsplit_block_size 2 and
> > -pc_fieldsplit_0 0,1.
> >
> > I was able to solve a problem with tens of millions dofs in 90 CG
> > iterations using these parameters.
> >
> > Best,
> > Jens
> >
> > On 01/31/2014 11:11 PM, David Knezevic wrote:
> >> On 01/31/2014 12:36 PM, John Peterson wrote:
> >>>> Hello,
> >>>>
> >>>> I'm trying to solve a linear elasticity problem based on a
> tetraheadral
> >>>> mesh with over a million degrees of freedom.
> >>>>
> >>>> I've been using the LU solver in MUMPS but I'm now forced to switch
> to an
> >>>> iterative solver because of the memory requirements of that solver.
> >>>>
> >>>> CG and BoomerAMG in hypre seem to be a good combination, but I've not
> been
> >>>> able to achieve convergence. The residual never goes below 1 even
> after
> >>>> several thousand iterations and shows high oscillations.
> >>>>
> >>>> The parameters I'm using are
> >>>> -ksp_type cg
> >>>> -ksp_monitor
> >>>> -ksp_converged_reason
> >>>> -pc_type hypre
> >>>> -pc_hypre_type boomeramg
> >>>> -pc_hypre_boomeramg_strong_threshold 0.7
> >>>> -log_summary
> >>>>
> >>> First thing I would think of is that the problem isn't actually
> symmetric
> >>> even though logically it should be?
> >>>
> >>> Does it work any better with -ksp_type bcgs?
> >>>
> >>>
> >>>
> >>>> -eps_monitor
> >>>> -st_type sinvert
> >>>>
> >>> Is this an eigenvalue problem too?
> >> Dana is working with me on this. It's not an eigenproblem, so
> >> "-eps_monitor -st_type sinvert" isn't meant to be there (but it those
> >> options would've just been ignored I believe).
> >>
> >> Regarding symmetry: That's a good thought. The PDE is linear elasticity,
> >> hence it's symmetric. But we enforce boundary conditions using
> >> DirchletBoundary, so that'll ruin symmetry on the Dirichlet rows/cols.
> >> Not sure if that's the problem here or not, but Dana will try "-ksp_type
> >> bcgs" to see if that helps convergence.
> >>
> >> Thanks,
> >> David
> >>
> >>
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> WatchGuard Dimension instantly turns raw network data into actionable
> >> security intelligence. It gives you real-time visual feedback on key
> >> security issues and trends.  Skip the complicated setup - simply import
> >> a virtual appliance and go from zero to informed in seconds.
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> >> _______________________________________________
> >> Libmesh-users mailing list
> >> Lib...@li...
> >> https://lists.sourceforge.net/lists/listinfo/libmesh-users
> >
> >
> ------------------------------------------------------------------------------
> > WatchGuard Dimension instantly turns raw network data into actionable
> > security intelligence. It gives you real-time visual feedback on key
> > security issues and trends.  Skip the complicated setup - simply import
> > a virtual appliance and go from zero to informed in seconds.
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Libmesh-users mailing list
> > Lib...@li...
> > https://lists.sourceforge.net/lists/listinfo/libmesh-users
>
>
>
> ------------------------------------------------------------------------------
> WatchGuard Dimension instantly turns raw network data into actionable
> security intelligence. It gives you real-time visual feedback on key
> security issues and trends.  Skip the complicated setup - simply import
> a virtual appliance and go from zero to informed in seconds.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> _______________________________________________
> Libmesh-users mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[Roy S. <roy...@ic...>]

On Sat, 1 Feb 2014, Jens Lohne Eftang wrote:

> I used these arguments for 3D elasticity and they seemed to work well
> also in combination with DirichletBoundary (why does that break symmetry?)

When you constrain your element matrix and vector (any constraints,
but including the heterogenous constraints from DirichletBoundary),
the default argument "asymmetric_constraint_rows=true" causes your
matrix to be left in an asymmetric state.  The advantage is that in
this case the linear solver can handle your constraint equations for
you; the disadvantage is an asymmetric state.
---
Roy

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[Jens L. E. <jle...@gm...>]

On 02/03/2014 05:18 PM, Roy Stogner wrote:
>
> On Sat, 1 Feb 2014, Jens Lohne Eftang wrote:
>
>> I used these arguments for 3D elasticity and they seemed to work well
>> also in combination with DirichletBoundary (why does that break 
>> symmetry?)
>
> When you constrain your element matrix and vector (any constraints,
> but including the heterogenous constraints from DirichletBoundary),
> the default argument "asymmetric_constraint_rows=true" causes your
> matrix to be left in an asymmetric state.  The advantage is that in
> this case the linear solver can handle your constraint equations for
> you; the disadvantage is an asymmetric state.
> ---
> Roy

With libmesh 0.9.3-pre, If I do use a DirichletBoundary (with a 
ZeroFunction or ConstFunction), and then after calling system.solve() 
loop over my system matrix entries to check symmetry, my matrix is 
symmetric to machine precision. And also the linear solver would have 
complained about any lack of symmetry (I would assume).

Has this default argument been changed in any later versions of libmesh?

Jens

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[Roy S. <roy...@ic...>]

On Mon, 3 Feb 2014, Jens Lohne Eftang wrote:

> With libmesh 0.9.3-pre, If I do use a DirichletBoundary (with a ZeroFunction 
> or ConstFunction), and then after calling system.solve() loop over my system 
> matrix entries to check symmetry, my matrix is symmetric to machine 
> precision.

What do the *constrain_elem* calls in your code look like?

Oh, wait, I'm being silly.  If you've only got constraints from
Dirichlet boundaries, then it doesn't matter whether you explicitly
enforce symmetry or not; the constraint equation "x_i = c" just has a
1 on the diagonal and no off-diagonal components.  It's only AMR
constraints that are likely to break symmetry.

> And also the linear solver would have complained about any lack of 
> symmetry (I would assume).

Not a safe assumption; testing for symmetry can be expensive.  I
recall once running into a preconditioner that would just silently
"converge" to the wrong answer if fed an asymmetric matrix.
---
Roy

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[John P. <jwp...@gm...>]

On Mon, Feb 3, 2014 at 8:26 AM, Dana Christen <dan...@ak...>wrote:

> Hi,
>
> Thanks for your assistance.
>
> Switching to BiCGStab (-ksp_type bcgs) doesn't improve convergence and
> shows the same symptoms (residual with frequent oscillations).
> Using fieldsplit as Jens suggests doesn't help either; residual doesn't go
> below 1e-3 even after 1000 iterations.
> Using GMRES makes the oscillations disappear, but the residual doesn't
> converge either.
>

OK, my other completely-anecdotally-based suggestion is that possibly the
AMG coarsening process removes enough constraints that the coarse-scale
problem has rigid body modes, thereby making the preconditioner singular?

If we light the Jed Brown signal fire by mentioning PETSc three times, he
might appear and explain why this suggestion is BS...

Also, since you said LU works for smaller problems, I don't suspect you
have an ill-posed problem, but have you checked to see how fast the
condition number is increasing with problem size?  If your problem is small
enough, you can run with:

-pc_type svd -pc_svd_monitor

And see what the condition number is.

-- 
John

Re: [Libmesh-users] BoomerAMG/hypre with CG for elasticity

[Dana C. <dan...@ak...>]

Our problem is too large to use SVD, but I believe it to be well-posed,
since it converges with LU and I've been able to make it converge using
CG+BoomerAMG using one of the example codes. We must have a problem
somewhere in our code somewhere...

Surprisingly I've noticed that PETSc doesn't use the -pc_fieldsplit_0
parameter:
Option left: name:-pc_fieldsplit_0 value: 0,1,2

Trying smoothed aggregation to include rigid body modes makes a lot of
sense. We may try that soon.

Dana


On Mon, Feb 3, 2014 at 6:53 PM, John Peterson <jwp...@gm...> wrote:

>
>
>
> On Mon, Feb 3, 2014 at 8:26 AM, Dana Christen <dan...@ak...>wrote:
>
>> Hi,
>>
>> Thanks for your assistance.
>>
>> Switching to BiCGStab (-ksp_type bcgs) doesn't improve convergence and
>> shows the same symptoms (residual with frequent oscillations).
>> Using fieldsplit as Jens suggests doesn't help either; residual doesn't go
>> below 1e-3 even after 1000 iterations.
>> Using GMRES makes the oscillations disappear, but the residual doesn't
>> converge either.
>>
>
> OK, my other completely-anecdotally-based suggestion is that possibly the
> AMG coarsening process removes enough constraints that the coarse-scale
> problem has rigid body modes, thereby making the preconditioner singular?
>
> If we light the Jed Brown signal fire by mentioning PETSc three times, he
> might appear and explain why this suggestion is BS...
>
> Also, since you said LU works for smaller problems, I don't suspect you
> have an ill-posed problem, but have you checked to see how fast the
> condition number is increasing with problem size?  If your problem is small
> enough, you can run with:
>
> -pc_type svd -pc_svd_monitor
>
> And see what the condition number is.
>
> --
> John
>