From: Vetter Roman <vetterro@et...>  20111129 17:52:08

Hi folks I'm using NewmarkSystem with PetscLinearSolver to solve M*a(t) + C*v(t) + K(t)*u(t) = f(t) in each time step. M and C are fixed, and the changes in K are very small (only 03 element contributions change between consecutive time steps). So, since I'm solving this millions of times for large systems in parallel, and the performance seems to be pretty bad when compared to other FEM software solving similar problems, I'm looking for a way to speed things up. The time dependence of K (and f for that matter) is absolutely minor, so I would like to reuse as many precomputed data as possible in the linear solver. Reusing the preconditioner and the current solution seems a good start. But there is probably lots of other work that can be skipped when almost the same system is solved again. What of the following is supported by libMesh, and how can I enable it?  Reusing the same preconditioner  Starting from the solution in the previous time step instead of zeros  Reusing other data in the solver, like decomposition matrices, etc.  Avoiding making copies of matrices, vectors, and destroying them again after the solve  Further optimizations for small changes? I'm using PETSc 3.2p5 and libMesh 0.7.2 (both in optimized mode, in case turning off debug mode for better performance comes to mind). I've been looking at PetscLinearSolver::solve() a bit and some of the mentioned optimizations seem at least possible, but it's also somewhat messy to get to understand what exactly is copied where and which preconditioner is used by default etc. from the various nested function calls. I'm hoping somebody who's more experienced with the interplay between libMesh and PETSc can give me a hint. I'm not using any fancy stuff like boundary conditions, subset solving or the like, just a plain bulk with slightly evolving stiffnesses and forces, that needs to be solved many many times. Any advice is much appreciated! Roman 
From: Vikram Garg <vikramvgarg@ma...>  20111129 18:07:35

Hey Vetter, There is an example of reusing the preconditioner in example 26. The steps are as follows: 1) Get the linear solver object: LinearSolver<Number> *linear_solver = system.get_linear_solver(); 2) Set the reuse preconditioner flag: linear_solver>reuse_preconditioner(bool flag_value); Hope that speeds things up a bit. Thanks. On Tue, Nov 29, 2011 at 11:51 AM, Vetter Roman <vetterro@...> wrote: > > Hi folks > > I'm using NewmarkSystem with PetscLinearSolver to solve M*a(t) + C*v(t) + K(t)*u(t) = f(t) in each time step. M and C are fixed, and the changes in K are very small (only 03 element contributions change between consecutive time steps). So, since I'm solving this millions of times for large systems in parallel, and the performance seems to be pretty bad when compared to other FEM software solving similar problems, I'm looking for a way to speed things up. The time dependence of K (and f for that matter) is absolutely minor, so I would like to reuse as many precomputed data as possible in the linear solver. Reusing the preconditioner and the current solution seems a good start. But there is probably lots of other work that can be skipped when almost the same system is solved again. > > What of the following is supported by libMesh, and how can I enable it? >  Reusing the same preconditioner >  Starting from the solution in the previous time step instead of zeros >  Reusing other data in the solver, like decomposition matrices, etc. >  Avoiding making copies of matrices, vectors, and destroying them again after the solve >  Further optimizations for small changes? > > I'm using PETSc 3.2p5 and libMesh 0.7.2 (both in optimized mode, in case turning off debug mode for better performance comes to mind). > > I've been looking at PetscLinearSolver::solve() a bit and some of the mentioned optimizations seem at least possible, but it's also somewhat messy to get to understand what exactly is copied where and which preconditioner is used by default etc. from the various nested function calls. I'm hoping somebody who's more experienced with the interplay between libMesh and PETSc can give me a hint. I'm not using any fancy stuff like boundary conditions, subset solving or the like, just a plain bulk with slightly evolving stiffnesses and forces, that needs to be solved many many times. > > Any advice is much appreciated! > > Roman > >  > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunknovd2d > _______________________________________________ > Libmeshusers mailing list > Libmeshusers@... > https://lists.sourceforge.net/lists/listinfo/libmeshusers  Vikram Garg PhD Candidate Institute for Computational and Engineering Sciences The University of Texas at Austin http://users.ices.utexas.edu/~vikram/ http://www.runforindia.org/runners/vikramg 
From: Derek Gaston <friedmud@gm...>  20111130 06:31:24

(I don't know much about the NewmarkSystem, so the following is targeted just at using a LinearImplicitSystem... But the ideas should be the same) Are you already separating out your M and C matrix? You can add those to your system by doing a system.add_matrix() call... Then assembling those two matrices once and then just do the matvecs each time. Similarly you can call system.add_vector() to add the a and v vectors. The value of the solution vector is automatically taken as the initial guess for the linear solver... So successive solves using the same LinearImplicitSystem should be doing what you want (along with reusing the preconditioner like Vikram mentioned). I'm not sure what you mean by "destroying them after the solve". If you don't destroy your EquationSystems object (ie delete it or let it go out of scope) then no matrices or vectors should be getting destroyed between solves. Similarly there are no copies of matrices / vectors (unless you are explicitly making them). If you know what entries are changing... Only update those entries! Derek Sent from my iPhone On Nov 29, 2011, at 11:52 AM, Vetter Roman <vetterro@...> wrote: > Hi folks > > I'm using NewmarkSystem with PetscLinearSolver to solve M*a(t) + C*v(t) + K(t)*u(t) = f(t) in each time step. M and C are fixed, and the changes in K are very small (only 03 element contributions change between consecutive time steps). So, since I'm solving this millions of times for large systems in parallel, and the performance seems to be pretty bad when compared to other FEM software solving similar problems, I'm looking for a way to speed things up. The time dependence of K (and f for that matter) is absolutely minor, so I would like to reuse as many precomputed data as possible in the linear solver. Reusing the preconditioner and the current solution seems a good start. But there is probably lots of other work that can be skipped when almost the same system is solved again. > > What of the following is supported by libMesh, and how can I enable it? >  Reusing the same preconditioner >  Starting from the solution in the previous time step instead of zeros >  Reusing other data in the solver, like decomposition matrices, etc. >  Avoiding making copies of matrices, vectors, and destroying them again after the solve >  Further optimizations for small changes? > > I'm using PETSc 3.2p5 and libMesh 0.7.2 (both in optimized mode, in case turning off debug mode for better performance comes to mind). > > I've been looking at PetscLinearSolver::solve() a bit and some of the mentioned optimizations seem at least possible, but it's also somewhat messy to get to understand what exactly is copied where and which preconditioner is used by default etc. from the various nested function calls. I'm hoping somebody who's more experienced with the interplay between libMesh and PETSc can give me a hint. I'm not using any fancy stuff like boundary conditions, subset solving or the like, just a plain bulk with slightly evolving stiffnesses and forces, that needs to be solved many many times. > > Any advice is much appreciated! > > Roman > >  > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunknovd2d > _______________________________________________ > Libmeshusers mailing list > Libmeshusers@... > https://lists.sourceforge.net/lists/listinfo/libmeshusers 
From: Vetter Roman <vetterro@et...>  20111220 09:13:36

Thank you guys for the suggestions and hints. The preconditioner helped a bit, and I'm looking into other stuff now. Derek, I was already separating out M and C, and only modified the parts in K that actually changed. Roman 