From: Roy S. <roy...@ic...> - 2012-10-26 23:32:50
Attachments:
parallel_sparsity.patch
|
Right now, we loop over all active elements when computing sparsity patterns. But with the attached patch, we loop over only active local elements, then we do a round robin communication to send relevant sparsity pattern row contributions to the processors which need them. Anyone have time to help me test this for correctness? It fixes the ParallelMesh problems I was aiming to fix, and I haven't hit any regressions yet with either ParallelMesh or SerialMesh, but it's such a deep change that I don't feel safe committing it based on just my preliminary tests. Testing for performance differences would be interesting, too. It would be easy enough to go back to a serial code path when the mesh isn't distributed, if the costs of looping over non-local elements turn out to be cheaper than the costs of trading results with other processors. --- Roy |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-10-27 00:42:37
|
I'll be happy to test this next week, but it is definitely the right move - especially as constraints become more expensive... Thinking about Paul's H1 and whatever else is coming... -Ben On Oct 26, 2012, at 6:32 PM, "Roy Stogner" <roy...@ic...> wrote: > > Right now, we loop over all active elements when computing sparsity > patterns. But with the attached patch, we loop over only active local > elements, then we do a round robin communication to send relevant > sparsity pattern row contributions to the processors which need them. > > Anyone have time to help me test this for correctness? It fixes the > ParallelMesh problems I was aiming to fix, and I haven't hit any > regressions yet with either ParallelMesh or SerialMesh, but it's such > a deep change that I don't feel safe committing it based on just my > preliminary tests. > > Testing for performance differences would be interesting, too. It > would be easy enough to go back to a serial code path when the mesh > isn't distributed, if the costs of looping over non-local elements > turn out to be cheaper than the costs of trading results with other > processors. > --- > Roy > <parallel_sparsity.patch> > ------------------------------------------------------------------------------ > WINDOWS 8 is here. > Millions of people. Your app in 30 days. > Visit The Windows 8 Center at Sourceforge for all your go to resources. > http://windows8center.sourceforge.net/ > join-generation-app-and-make-money-coding-fast/ > _______________________________________________ > Libmesh-devel mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-devel |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-10-27 01:26:19
|
You said sparsity patterns, I read constraints... Still, I'll be happy to test it. -Ben On Oct 26, 2012, at 6:32 PM, "Roy Stogner" <roy...@ic...> wrote: > > Right now, we loop over all active elements when computing sparsity > patterns. But with the attached patch, we loop over only active local > elements, then we do a round robin communication to send relevant > sparsity pattern row contributions to the processors which need them. > > Anyone have time to help me test this for correctness? It fixes the > ParallelMesh problems I was aiming to fix, and I haven't hit any > regressions yet with either ParallelMesh or SerialMesh, but it's such > a deep change that I don't feel safe committing it based on just my > preliminary tests. > > Testing for performance differences would be interesting, too. It > would be easy enough to go back to a serial code path when the mesh > isn't distributed, if the costs of looping over non-local elements > turn out to be cheaper than the costs of trading results with other > processors. > --- > Roy > <parallel_sparsity.patch> > ------------------------------------------------------------------------------ > WINDOWS 8 is here. > Millions of people. Your app in 30 days. > Visit The Windows 8 Center at Sourceforge for all your go to resources. > http://windows8center.sourceforge.net/ > join-generation-app-and-make-money-coding-fast/ > _______________________________________________ > Libmesh-devel mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-devel |
From: Roy S. <roy...@ic...> - 2012-10-27 05:10:04
|
On Fri, 26 Oct 2012, Kirk, Benjamin (JSC-EG311) wrote: > You said sparsity patterns, I read constraints... Still, I'll be happy to test it. You might have to wait, sadly - I've already managed to trigger two bugs with it. One is a plain dumb "I forgot about the MPI+threading hybrid case" problem that will be tedious but straight forward to fix; the other I don't understand yet and could take anywhere from minutes to days. I'll post again when I've got a patch I can't break. --- Roy |
From: Derek G. <fri...@gm...> - 2012-10-27 15:53:43
|
Keep in mind the ability to add extra stuff to the sparsity pattern (using AugmentSparsityPattern). That is something we rely on. Also.... what is the current state of libMesh svn? I just got this compile error: Compiling C++ (in optimized mode) src/base/dof_map_constraints.C... src/base/dof_map_constraints.C: In member function ‘void libMesh::DofMap::scatter_constraints(libMesh::MeshBase&)’: src/base/dof_map_constraints.C:2875: error: ‘mesh’ was not declared in this scope make: *** [src/base/dof_map_constraints.x86_64-apple-darwin11.4.2.opt.o] Error 1 Is it me - or did I happen to checkout a bad revision? Derek On Fri, Oct 26, 2012 at 11:09 PM, Roy Stogner <roy...@ic...>wrote: > > On Fri, 26 Oct 2012, Kirk, Benjamin (JSC-EG311) wrote: > > > You said sparsity patterns, I read constraints... Still, I'll be happy > to test it. > > You might have to wait, sadly - I've already managed to trigger two > bugs with it. One is a plain dumb "I forgot about the MPI+threading > hybrid case" problem that will be tedious but straight forward to fix; > the other I don't understand yet and could take anywhere from minutes > to days. > > I'll post again when I've got a patch I can't break. > --- > Roy > > > ------------------------------------------------------------------------------ > WINDOWS 8 is here. > Millions of people. Your app in 30 days. > Visit The Windows 8 Center at Sourceforge for all your go to resources. > http://windows8center.sourceforge.net/ > join-generation-app-and-make-money-coding-fast/ > _______________________________________________ > Libmesh-devel mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-devel > |
From: Roy S. <roy...@ic...> - 2012-10-27 16:09:46
|
On Sat, 27 Oct 2012, Derek Gaston wrote: > Keep in mind the ability to add extra stuff to the sparsity pattern > (using AugmentSparsityPattern). That is something we rely on. Hmm... how do you want to handle extra sparsity entries in the distributed case? 1. User code is responsible for correctly adding all extra entries to each processor's sparsity rows on that processor. This should still work fine now. 2. User code can add extra remote sparsity entries to the new nonlocal part of the sparsity pattern, and then they get passed on to the right processors in the communications step. This would be work after I move one line. You know, supporting (2) wouldn't actually add anything to computational cost and wouldn't interfere with support for (1). I'm leaning toward (2) now. ;-) > src/base/dof_map_constraints.C:2875: error: ‘mesh’ was not declared in this scope > make: *** [src/base/dof_map_constraints.x86_64-apple-darwin11.4.2.opt.o] Error 1 Should be fixed in r6227, thanks. --- Roy |
From: Derek G. <fri...@gm...> - 2012-10-27 16:13:02
|
On Sat, Oct 27, 2012 at 10:09 AM, Roy Stogner <roy...@ic...>wrote: > 1. User code is responsible for correctly adding all extra entries to > each processor's sparsity rows on that processor. This should still > work fine now. > > 2. User code can add extra remote sparsity entries to the new > nonlocal part of the sparsity pattern, and then they get passed on to > the right processors in the communications step. This would be work > after I move one line. > > You know, supporting (2) wouldn't actually add anything to > computational cost and wouldn't interfere with support for (1). I'm > leaning toward (2) now. ;-) (2) is fine with me... We actually don't compute off-processor extra sparsity entries currently. We do all the work to get all the info on the local processor that the local processor needs to make decisions about it's sparsity pattern and then just add the local entries. I might rethink some of that code if there is some new capability now... Derek |
From: Roy S. <roy...@ic...> - 2012-10-27 16:36:08
|
On Sat, 27 Oct 2012, Derek Gaston wrote: > We actually don't compute off-processor extra sparsity entries > currently. We do all the work to get all the info on the local > processor that the local processor needs to make decisions about > it's sparsity pattern and then just add the local entries. Really? That can be hard as hell. I wrote this communication code because I hit a case where DoF A on proc X was being contrained indirectly in terms of DoF B on proc Y, and even after making sure proc Y and X both knew about the constraint so Y could add the (B-stencil,A) entries correctly, there was no way to get proc X to add the corresponding (A,B-stencil) entries, because the elements supporting B weren't all semilocal on X. > I might rethink some of that code if there is some new capability now... Well, let's wait until the new capability is working. ;-) The first bug I triggered with it turned out to be unrelated and easy-to-fix, the next one looks to be unrelated and not-too-hard-to-fix, the threading+MPI bug ought to be tedious-but-easy to fix... but we're starting to talk about a lot of fixes here. --- Roy |
From: Derek G. <fri...@gm...> - 2012-10-27 16:29:56
|
On Sat, Oct 27, 2012 at 10:09 AM, Roy Stogner <roy...@ic...>wrote: > src/base/dof_map_constraints.**C:2875: error: ‘mesh’ was not declared in > this scope > >> make: *** [src/base/dof_map_constraints.**x86_64-apple-darwin11.4.2.opt.* >> *o] Error 1 >> > > Should be fixed in r6227, thanks. > Hmmm... I'm still getting this: src/base/dof_map_constraints.C: In member function ‘void libMesh::DofMap::scatter_constraints(libMesh::MeshBase&)’: src/base/dof_map_constraints.C:2871: error: ‘mesh’ was not declared in this scope make: *** [src/base/dof_map_constraints.x86_64-apple-darwin11.4.2.opt.o] Error 1 |
From: Roy S. <roy...@ic...> - 2012-10-27 16:41:06
|
On Sat, 27 Oct 2012, Derek Gaston wrote: > Hmmm... I'm still getting this: > > src/base/dof_map_constraints.C: In member function ‘void libMesh::DofMap::scatter_constraints(libMesh::MeshBase&)’: > src/base/dof_map_constraints.C:2871: error: ‘mesh’ was not declared in this scope > make: *** [src/base/dof_map_constraints.x86_64-apple-darwin11.4.2.opt.o] Error 1 Damn, did I un-ifdef the wrong argument? Try r6228. And then if that works try r6228 with r6227 reverted... Sorry for the mess; I don't have any --disable-node-constraints builds handy except the one in our BuildBot checks, and that only gets run once or twice a day. --- Roy |
From: Derek G. <fri...@gm...> - 2012-10-28 16:33:55
|
On Sat, Oct 27, 2012 at 10:41 AM, Roy Stogner <roy...@ic...>wrote: > Sorry for the mess; > Meh - it happens. It appears to be compiling today ;-) Derek |
From: Roy S. <roy...@ic...> - 2012-10-29 20:25:17
Attachments:
parallel_sparsity.patch
|
On Sat, 27 Oct 2012, Roy Stogner wrote: > I'll post again when I've got a patch I can't break. Not quite "a patch I can't break" yet, so don't waste time doing any non-automated testing, but here's "a patch I can't break quite as easily" if anyone else wants to run it through a gauntlet on the configurations I can't break myself. It seems to work for SerialMesh with or without threads and ParallelMesh without, but still breaks pretty easily when run threaded on a ParallelMesh. --- Roy |
From: Roy S. <roy...@ic...> - 2012-11-05 18:32:57
Attachments:
parallel_sparsity.patch
|
On Mon, 29 Oct 2012, Roy Stogner wrote: > On Sat, 27 Oct 2012, Roy Stogner wrote: > >> I'll post again when I've got a patch I can't break. Here goes. The attached patch seems to work for me with both SerialMesh and ParallelMesh, and doesn't break with threading this time. Interestingly enough, the *svn head* code does seem to be breakable (under very unlikely circumstances) with threading - it seems to be erroneously assuming that Body::operator()(Range&) will only be called once within any particular thread. If anyone else finds flaws in this patch, then before going back to the drawing board I'll want to figure out a separate smaller fix for that bug. --- Roy |