From: Lorenzo A. B. <bot...@gm...> - 2012-08-29 08:48:41
|
Hi Lei, I'll try to answer to some of your questions since I wrote this dG example. In general it is not easy to identify which is the most expensive portion of a code and libMesh developers make extensive use of profiling tools. Something might be inefficient in your specific application but efficiency concerns imply that you have something to compare with. > > In this example, the Laplace equation is solved using DG with interior penalty method. So the main job is to compute the mass matrix from the element volume integration and surface jump integration. It is done element by element with the help of element iterator. I know all of the elements organized as a quad-forest and only the active items need to be looped. I guess you mean stiffness matrix. > > so does this mean when we need to reach to the next active element, const_element_iterator++ need to go through the quad-forest and pass several non-active parent elements then get to the correct element pointer to the next active element? If the answer is true. Is it too expensive to find an element in a solver? > > The other thing is the metrics computing such as Jacobian, xi_x xi_y eta_x eta_y and face normal etc. Do those metrics computed several times using FE::FEMap? If the mesh is fixed in space, we only need to compute them once and store them somewhere. > > Same thing happens when computing quadrature points in physical domain and value of shape function/ its derivation at those points in computational domain. I think those are computed in > ---------------------- > fe->reinit(elem) > ---------------------- > Those values do not change during the calculation and should be computed only once. I think they are in the deepest loop and will eat a lot of cpu time. > You can always store quantities that you need several times but this might eat up a lot of memory. Accessing to memory might be expensive and sometimes it might be even faster to recompute. The libMesh approach is to recompute everything. In your application you might store something but, when the problem you need to solve reaches a critical size, you want to store quantities that are really expensive to compute. I think that one important point is time integration. In explicit solvers assembly times dominates and it might be useful to store FE related quantities. In implicit solvers usually the memory is used to store the global system matrix, the preconditioner ecc... > Last one, to compute the flux jump at flux points on the common surface of two cells with different h levels. The coordinates and normals of the flux quadrature points on the both side of element are computed several times if we use a residual based scheme. It does not change at each refinement level. Is there any efficient way to handle those kind of situation: flux jump calculation at a surface with hanging node? When you do h-refinement the calculation of interface quantities is actually more expensive since you need to find matching quadrature points on the sides of neighboring elements. If you use reference frame basis functions starting from a physical frame quadrature point you need to find the corresponding reference frame quadrature point on the neighbor. This requires to invert the reference to physical frame map by means of the Newton method. > > Or those are just for demo only. Those efficiency problem get even worse for a scheme need residual jacobian at each time step. Do we need to use FEMap explicitly and store the metrics for each element/face? So we don't need to compute them more than once. This is just a demo that make sense for the Laplace equation and it shows how to compute the quantities you need. If you want to store quantities you will need to care care of that. > > Thanks a lot for your time. Greetings Lorenzo > > > > Sincerely Yours, > > Lei Shi > ---------- > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ > Libmesh-devel mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-devel |
From: Roy S. <roy...@ic...> - 2012-08-29 16:56:31
|
On Wed, 29 Aug 2012, Lorenzo Alessio Botti wrote: > You can always store quantities that you need several times but this > might eat up a lot of memory. Accessing to memory might be expensive > and sometimes it might be even faster to recompute. This can absolutely be true - on modern processors some problems end up, even in the assembly, bottlenecked by memory bandwidth rather than CPU speed. At that point anything that you can compute based on data that's already in the processor cache is "free". > The libMesh approach is to recompute everything. That's the libMesh examples approach, anyway, but that's intended to keep the examples simple more than anything else, I thought. When profiling real applications I've found that finding ways to cache transcendental function evaluations (e.g. by doing nodal quadrature or reinterpolation of such terms) can give a decent savings in runtime. Of course efficiency discussion is influenced by the fact that all the main libMesh developers are profiling based on our own app codes: implicit solves with intricate constitutive models. If you're doing an explicit calculation then it probably makes sense to cache even mappings and shape functions; if you're solving a linear problem then you could do much better with Rob Kirby's element matrix transformation tricks than with plain quadrature; etc. --- Roy |
From: Derek G. <fri...@gm...> - 2012-08-29 17:04:57
|
With MOOSE we have the full gamut of options. The default is to recompute everything. We also have the option to cache one FE reinit in the case where you have a regular grid. We also have the option of caching EVERY FE reinit for every element so that you can reuse them until the mesh changes. I'll say that that last one is not used very often... it eats up SO much damn memory... especially in the cases where it would be useful (like on higher order elements in 3D). So it's kind of a self-defeating optimization... Derek Sent from my iPhone On Aug 29, 2012, at 5:56 PM, Roy Stogner <roy...@ic...> wrote: > > On Wed, 29 Aug 2012, Lorenzo Alessio Botti wrote: > >> You can always store quantities that you need several times but this >> might eat up a lot of memory. Accessing to memory might be expensive >> and sometimes it might be even faster to recompute. > > This can absolutely be true - on modern processors some problems end > up, even in the assembly, bottlenecked by memory bandwidth rather than > CPU speed. At that point anything that you can compute based on data > that's already in the processor cache is "free". > >> The libMesh approach is to recompute everything. > > That's the libMesh examples approach, anyway, but that's intended to > keep the examples simple more than anything else, I thought. When > profiling real applications I've found that finding ways to cache > transcendental function evaluations (e.g. by doing nodal quadrature or > reinterpolation of such terms) can give a decent savings in runtime. > > Of course efficiency discussion is influenced by the fact that all the > main libMesh developers are profiling based on our own app codes: > implicit solves with intricate constitutive models. If you're doing > an explicit calculation then it probably makes sense to cache even > mappings and shape functions; if you're solving a linear problem then > you could do much better with Rob Kirby's element matrix > transformation tricks than with plain quadrature; etc. > --- > Roy > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Libmesh-users mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-users |
From: Jed B. <je...@59...> - 2012-08-29 17:52:56
|
I'll also comment that aggressive caching is pessimal from a modern-hardware and energy perspective. Memory and memory bandwidth take an increasing part of the power and acquisition budget. On modern hardware, for operations that can be vectorized, you should usually plan to recompute everything that takes less than 50 flops per scalar value. If you have a computation that takes more than 50 flops to recompute, you may want to store it, but be aware that reloading it may displace more useful cache lines and if you aren't careful about cache locality (e.g. if the element is visited by a thread running on a different socket later), performance results may be very bad and/or not reproducible. Lei, I suggest being very conservative when selecting what may be worth caching. Also, depending on your application, there may be much larger gains to be had by looking elsewhere. On Wed, Aug 29, 2012 at 12:04 PM, Derek Gaston <fri...@gm...> wrote: > With MOOSE we have the full gamut of options. The default is to > recompute everything. We also have the option to cache one FE reinit > in the case where you have a regular grid. We also have the option of > caching EVERY FE reinit for every element so that you can reuse them > until the mesh changes. > > I'll say that that last one is not used very often... it eats up SO > much damn memory... especially in the cases where it would be useful > (like on higher order elements in 3D). So it's kind of a > self-defeating optimization... > > Derek > > Sent from my iPhone > > On Aug 29, 2012, at 5:56 PM, Roy Stogner <roy...@ic...> wrote: > > > > > On Wed, 29 Aug 2012, Lorenzo Alessio Botti wrote: > > > >> You can always store quantities that you need several times but this > >> might eat up a lot of memory. Accessing to memory might be expensive > >> and sometimes it might be even faster to recompute. > > > > This can absolutely be true - on modern processors some problems end > > up, even in the assembly, bottlenecked by memory bandwidth rather than > > CPU speed. At that point anything that you can compute based on data > > that's already in the processor cache is "free". > > > >> The libMesh approach is to recompute everything. > > > > That's the libMesh examples approach, anyway, but that's intended to > > keep the examples simple more than anything else, I thought. When > > profiling real applications I've found that finding ways to cache > > transcendental function evaluations (e.g. by doing nodal quadrature or > > reinterpolation of such terms) can give a decent savings in runtime. > > > > Of course efficiency discussion is influenced by the fact that all the > > main libMesh developers are profiling based on our own app codes: > > implicit solves with intricate constitutive models. If you're doing > > an explicit calculation then it probably makes sense to cache even > > mappings and shape functions; if you're solving a linear problem then > > you could do much better with Rob Kirby's element matrix > > transformation tricks than with plain quadrature; etc. > > --- > > Roy > > > > > ------------------------------------------------------------------------------ > > Live Security Virtual Conference > > Exclusive live event will cover all the ways today's security and > > threat landscape has changed and how IT managers can respond. Discussions > > will include endpoint security, mobile security and the latest in malware > > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > > _______________________________________________ > > Libmesh-users mailing list > > Lib...@li... > > https://lists.sourceforge.net/lists/listinfo/libmesh-users > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Libmesh-users mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-users > |