From: John P. <pet...@cf...> - 2007-07-13 14:37:55
|
Roy Stogner writes: > On Thu, 12 Jul 2007, John Peterson wrote: > > > I think this definitely takes us in the right direction, though I don't > > doubt there will be several gotchas even with this first step. We should > > also think about meshes which can't fit on a single processor... > > > > Such a mesh would be ungainly (how would we store it, multiple > > data files?) but you can certainly build_square() something which > > is too large for a single CPU, and it would be cool if that eventually > > worked in parallel w/o ever having to store the whole thing on 1 CPU. > > Absolutely. Oh, and I forgot to mention, even if your mesh *is* small enough to fit on 1 CPU (i.e. you can successfully open the mesh file and read it all in) it may be too big for 16 copies to fit, supposing you are running on some kind of quad-quad box. I don't think fewer cores are coming back in style any time soon. > I think Ben's got the right idea about storage (even though it makes > Mesh::read() tricky, since we'd have to make multiple passes to avoid > reading in too many elements/nodes at once). What worries me about > meshes too big to fit on one node is repartitioning. Will Parmetis > work that way? The space-filling-curve repartitioners might be ideal > for use in parallel; putting all the elements in some 1D order is easy > to do in parallel, and then it's easy to negotiate which processors > get which elements. I believe parmetis is designed to handle partitioning in parallel (based on the name only, I haven't tried it). So it should work if we read in a chunk of elements, send that out to a processor, and repeat until we've read them all in. Then a "real" partitioner could partition them correctly, incurring additional communication overhead of course. We also implemented a parallel sort algorithm some years back (I have this code somewhere, I'm sure) which would probably allow the space-filling curve partitioning you suggest. -John |