From: Tim K. <tim...@ce...> - 2008-04-16 15:02:04
|
Dear libMesh team, As I'm just so much in asking questions, here's another one: How will the performance of EquationSystems::reinit() behave when ParallelMesh is ready for use by user application? The reason for my question is that for my code on a large number of processors, EquationSystems::reinit() is quite a bottle neck. As far as I understand function does mainly two things: It projects the solutions (which can be done mostly locally, i.e. should scale well) and then redistributes them (which seem to scale badly). The question thus is whether redistribution the solution vectors (and the mesh) is faster with parallel mesh. Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim...@me..., tim...@ce... Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen |
From: Roy S. <ro...@st...> - 2008-04-16 15:09:33
|
On Wed, 16 Apr 2008, Tim Kroeger wrote: > As I'm just so much in asking questions, here's another one: How will > the performance of EquationSystems::reinit() behave when ParallelMesh > is ready for use by user application? At first, just as badly as before. System::project_vector() is still one of the things we use a serial vector for, and while that's on the list of ParallelMesh items to be fixed, it's not high enough on the list. > The reason for my question is that for my code on a large number of > processors, EquationSystems::reinit() is quite a bottle neck. You're not the only one seeing that. We'll definitely fix the poor scaling there before we consider ParallelMesh "done", but there will probably be a 0.6.9 release first once ParallelMesh is in a "correct but not yet complete" state. --- Roy |
From: Tim K. <tim...@ce...> - 2008-08-26 13:43:38
|
Dear libMesh developer team, Could someone please tell me what the current state about the item mentioned below is? I just want to know, so that I can decide whether I should wait or get around somehow or whatever. (In the current state, my code does not show any speedup on 20 processors compared with 1 processor, because it spends apparently 95% of the time in EquationSystems::reinit().) Best Regards, Tim On Wed, 16 Apr 2008, Roy Stogner wrote: > > On Wed, 16 Apr 2008, Tim Kroeger wrote: > >> As I'm just so much in asking questions, here's another one: How will >> the performance of EquationSystems::reinit() behave when ParallelMesh >> is ready for use by user application? > > At first, just as badly as before. System::project_vector() is still > one of the things we use a serial vector for, and while that's on the > list of ParallelMesh items to be fixed, it's not high enough on the > list. > >> The reason for my question is that for my code on a large number of >> processors, EquationSystems::reinit() is quite a bottle neck. > > You're not the only one seeing that. We'll definitely fix the poor > scaling there before we consider ParallelMesh "done", but there will > probably be a 0.6.9 release first once ParallelMesh is in a "correct > but not yet complete" state. > --- > Roy > -- Dr. Tim Kroeger Phone +49-421-218-7710 tim...@me..., tim...@ce... Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen |
From: Tim K. <tim...@ce...> - 2008-08-27 10:21:43
|
Dear Roy, On Tue, 26 Aug 2008, Roy Stogner wrote: >> (In the current state, my code does not show any speedup on 20 processors >> compared with 1 processor, because it spends apparently 95% of the time in >> EquationSystems::reinit().) > > Could you insert some more fine-grained performance logs and verify > that it's really project_vector causing most of the delay? That's an > astonishingly lousy result. Well, the '95%' were just the sensed percentage. To find out how much it really is, I now used the PerfLog class. I have to admit that I had not used that class ever before, so I am not completely sure whether I understood the concept correctly. I did the following: In EquationSystems::reinit(), I added some PerfLog stuff (see attached modified copy of equation_systmes.C). Also, in my main program, I added a PerfLog instance that measures the complete computation time. Attached are the results for 1 cpu and for 20 cpus. Note that EquationSystems::reinit() has been called 6 times, hence there are 6 sections about this. (This is where I'm not sure whether I understood the concept correctly.) Note that the total time the computation spends in this function is about 220 seconds for 1 cpu and about 230 seconds for 20 cpus, whereas the overall total computation time is about 660 seconds for 1 cpu and about 345 seconds for 20 cpus. Hence, the speedup is less than a factor of 2. Also, for 20 cpus, 67% of the time is spent in EquationSystmes::reinit(). I hope this information is sufficient for you. If not, please let me know what else you need. By the way: After the last refinement, the mesh consists of 98897 elements and 22675 nodes for 1 cpu, 98841 elements and 22662 nodes for 20 cpus. Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim...@me..., tim...@ce... Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen |