From: Cody P. <cod...@gm...> - 2013-02-13 20:05:18
|
One of the things (maybe the only thing) that we are missing when broadcasting the mesh are the boundary/subdomain names which are stored in boundary_info. Namely these two guys: std::map<boundary_id_type, std::string> _ss_id_to_name; std::map<boundary_id_type, std::string> _ns_id_to_name; I have only myself to blame since I originally added these data structures and to this date they are still only populated when using the Exodus reader. We've become rather found of this capability in MOOSE and would like to get this data sent along with all the rest of the data during the broadcast operation. In fact, right now we are using a crutch and skipping mesh.read() and loading the mesh on every processor as a workaround so that we can retrieve that information on each processor. Yeah... this needs to get fixed. So I'd be happy to work on this unless someone else wants to volunteer. Clearly we'll need a pack this data up since the strings will be variable length as we send this over the wire. I might need a little guidance as to where or how this can be done. I looked at the mesh_broadcast routine and can't immediately see where all of the boundary_info data is sent. I'm missing something here... Thanks, Cody |
From: Roy S. <roy...@ic...> - 2013-02-13 20:27:32
|
On Wed, 13 Feb 2013, Cody Permann wrote: > So I'd be happy to work on this unless someone else wants to > volunteer. Clearly we'll need a pack this data up since the strings > will be variable length as we send this over the wire. I might need > a little guidance as to where or how this can be done. I looked at > the mesh_broadcast routine and can't immediately see where all of > the boundary_info data is sent. I'm missing something here... The trick is that boundary info is packed up along with the elements and nodes it applies to. See parallel_elem.C and parallel_node.C for the implementation. We don't want every processor to know about every boundary id, only for every id on its own part of the boundary. I don't think this'll be what we want to do for those maps you added, though. Those ought to be serialized, right? It would save a little memory but add a lot of communication if we tried to have each rank only store labels corresponding to boundary ids that had some support on that rank's partition. I'd love to add Parallel::pack/unpack/etc. implementations for std::pair and std::string. Then you'd be able to broadcast each whole map with a single function call. I'm not sure when I'd find time to get to it, though, so if you definitely want it before March you'll have to do it mostly yourself; ping me with any questions. --- Roy |
From: Cody P. <cod...@gm...> - 2013-02-13 20:43:41
|
On Wed, Feb 13, 2013 at 1:27 PM, Roy Stogner <roy...@ic...>wrote: > > On Wed, 13 Feb 2013, Cody Permann wrote: > > So I'd be happy to work on this unless someone else wants to >> volunteer. Clearly we'll need a pack this data up since the strings >> will be variable length as we send this over the wire. I might need >> a little guidance as to where or how this can be done. I looked at >> the mesh_broadcast routine and can't immediately see where all of >> the boundary_info data is sent. I'm missing something here... >> > > The trick is that boundary info is packed up along with the elements > and nodes it applies to. See parallel_elem.C and parallel_node.C for > the implementation. > We don't want every processor to know about every > boundary id, only for every id on its own part of the boundary. > > I don't think this'll be what we want to do for those maps you added, > though. Hmm, so which is it? If you don't want ids everywhere, why would you want strings everywhere? > Those ought to be serialized, right? It would save a little > memory but add a lot of communication if we tried to have each rank > only store labels corresponding to boundary ids that had some support > on that rank's partition. > Ah, so we have to decide on which trade-off we want... > > I'd love to add Parallel::pack/unpack/etc. implementations for > std::pair and std::string. Then you'd be able to broadcast each whole > map with a single function call. std::string for sure, but std::pair? Why not add an implementation for std::map that just send pairs of vectors? I'm sure there are lots of trade-offs for all of these designs. > I'm not sure when I'd find time to > get to it, though, so if you definitely want it before March you'll > have to do it mostly yourself; ping me with any questions. > I guess we'll see - I don't know if we need this right now or not. We have a huge backlog all the time so we just have to decide what gets done and what doesn't, I'm sure you know the feeling. Cody > --- > Roy |
From: Roy S. <roy...@ic...> - 2013-02-13 21:08:21
|
On Wed, 13 Feb 2013, Cody Permann wrote: > We don't want every processor to know about every > boundary id, only for every id on its own part of the boundary. > > I don't think this'll be what we want to do for those maps you added, > though. > > Hmm, so which is it? If you don't want ids everywhere, why would you want strings everywhere? It's not "value==boundary_id_type" vs "value==string" that I was thinking about, it's "key==Node*" (or Elem*,side) vs "key==boundary_id_type". The boundary ids are indexed by node and elem; i.e. on a typical mesh with a million elements and a dozen side sets, you've got tens of thousands of ids to worry about, and the number keeps growing with the size of the coarse mesh. The boundary names are indexed by id; i.e. on that same mesh you've got a dozen strings to pass around, and the number is fixed for a given BVP. > Those ought to be serialized, right? It would save a little > memory but add a lot of communication if we tried to have each rank > only store labels corresponding to boundary ids that had some support > on that rank's partition. > > Ah, so we have to decide on which trade-off we want... Right. But now that I think about it longer: communications efficiency means we definitely wouldn't want to stick boundary id names into the Node/Elem packing, but that means we have to send the boundary id names separately, and so we could always get the set of boundary ids that are supported on a given set of Nodes/Elems and then send just those names. Combine that with a deletion step when removing nonsemilocal elements and there's no reason it would have to be inefficient to keep the boundary names fully parallelized. > I'd love to add Parallel::pack/unpack/etc. implementations for > std::pair and std::string. > > Then you'd be able to broadcast each whole > map with a single function call. > > std::string for sure, but std::pair? Why not add an implementation > for std::map that just send pairs of vectors? If we do std::pair, then we get std::map automatically, and we also get multimap, unordered_map/multimap, GCC's hash versions, Google's btree versions, whatever, for free. We can also often do fancier things when passing iterators than we can when passing containers. This is why much of the mesh redistribution code is so simple now - for example, if we need to send elements in ascending order by level so as to build up the parent pointers correctly, we don't have to copy each set into containers of Elem pointers first, we just pass the same predicate-iterators that libMesh already provides into packed_range Parallel:: routines. > I guess we'll see - I don't know if we need this right now or not. > We have a huge backlog all the time so we just have to decide what > gets done and what doesn't, I'm sure you know the feeling. Indeed, this sentence makes me want to laugh and cry all at once. --- Roy |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2013-02-13 21:13:46
|
On Feb 13, 2013, at 3:08 PM, Roy Stogner <roy...@ic...> wrote: > Combine that with a deletion step when > removing nonsemilocal elements and there's no reason it would have to > be inefficient to keep the boundary names fully parallelized. But in the interest of keeping it simple too, you might look at Communicator::set_union std::map<T1,T2> &data) … at line 1824 of parallel_implementation.h You can broadcast the parallelized boundary names simply by calling this guy, and implementing whatever may be needed for your std::string. -Ben |
From: Roy S. <roy...@ic...> - 2013-02-13 21:18:00
|
On Wed, 13 Feb 2013, Kirk, Benjamin (JSC-EG311) wrote: > Communicator::set_union std::map<T1,T2> &data) … > > at line 1824 of parallel_implementation.h > > You can broadcast the parallelized boundary names simply by calling > this guy, and implementing whatever may be needed for your > std::string. Can this work? That method (and pretty much everything else except the new packed_range stuff) was designed for arbitrary sized containers but with fixed sizes in mind for the underlying data types, I'm afraid. We have a bunch of specializations for individual strings (where the underlying data type is a fixed-width char or wide char) but not so much for containers of strings. --- Roy |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2013-02-13 21:23:27
|
On Feb 13, 2013, at 3:17 PM, Roy Stogner <roy...@ic...> wrote: > On Wed, 13 Feb 2013, Kirk, Benjamin (JSC-EG311) wrote: > >> Communicator::set_union std::map<T1,T2> &data) … >> >> at line 1824 of parallel_implementation.h >> >> You can broadcast the parallelized boundary names simply by calling >> this guy, and implementing whatever may be needed for your >> std::string. > > Can this work? That method (and pretty much everything else except > the new packed_range stuff) was designed for arbitrary sized > containers but with fixed sizes in mind for the underlying data types, > I'm afraid. We have a bunch of specializations for individual strings > (where the underlying data type is a fixed-width char or wide char) > but not so much for containers of strings. out of the box I doubt it, but putting your container in there and following the rabbit trail of T2 would be instructive as to what would be required is all I mean. -Ben |
From: Roy S. <roy...@ic...> - 2013-02-13 21:30:50
|
On Wed, 13 Feb 2013, Roy Stogner wrote: > On Wed, 13 Feb 2013, Kirk, Benjamin (JSC-EG311) wrote: > >> Communicator::set_union std::map<T1,T2> &data) … >> >> at line 1824 of parallel_implementation.h >> >> You can broadcast the parallelized boundary names simply by calling >> this guy, and implementing whatever may be needed for your >> std::string. > > Can this work? After some thought, I can answer my own question: "yes, but not quite yet". For containers which aren't contiguous blocks of memory, we have to pack things up into contiguous blocks for MPI already. Rather than having separate types of code to do this for fixed-width and variable-width types, we could easily just use the variable-width case all the time, with some simple compile-time metaprogramming tricks to define "pack", "unpack", etc. for any objects which already have a Parallel::StandardType<T> specialization. But "not quite yet" might be an understatement; even if this conversation suckers me into doing the pair/string packing quickly, getting it to work with the originally fixed-size-only APIs would be a longer-term challenge. --- Roy |
From: Derek G. <fri...@gm...> - 2013-02-13 21:36:26
|
How about just finding the max length for the names and reading all the names into fixed length char*s.... and just using the available routines to send them across the wire. We don't need to over think this... we are sending some strings across the wire once per simulation...... Derek On Wed, Feb 13, 2013 at 2:30 PM, Roy Stogner <roy...@ic...>wrote: > > On Wed, 13 Feb 2013, Roy Stogner wrote: > > On Wed, 13 Feb 2013, Kirk, Benjamin (JSC-EG311) wrote: >> >> Communicator::set_union std::map<T1,T2> &data) … >>> >>> at line 1824 of parallel_implementation.h >>> >>> You can broadcast the parallelized boundary names simply by calling >>> this guy, and implementing whatever may be needed for your >>> std::string. >>> >> >> Can this work? >> > > After some thought, I can answer my own question: "yes, but not quite > yet". For containers which aren't contiguous blocks of memory, we > have to pack things up into contiguous blocks for MPI already. Rather > than having separate types of code to do this for fixed-width and > variable-width types, we could easily just use the variable-width case > all the time, with some simple compile-time metaprogramming tricks to > define "pack", "unpack", etc. for any objects which already have a > Parallel::StandardType<T> specialization. > > But "not quite yet" might be an understatement; even if this > conversation suckers me into doing the pair/string packing quickly, > getting it to work with the originally fixed-size-only APIs would be a > longer-term challenge. > --- > Roy > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > Libmesh-devel mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-devel > > |
From: Roy S. <roy...@ic...> - 2013-02-13 21:49:32
|
On Wed, 13 Feb 2013, Derek Gaston wrote: > How about just finding the max length for the names and reading all > the names into fixed length char*s.... and just using the available > routines to send them across the wire. That would definitely work. > We don't need to over think this... we are sending some strings > across the wire once per simulation...... Once per repartitioning, at most. But yeah, I'm not overthinking this because infrequently synchronizing a few strings is still hard; rather because taking time to overthink Parallel:: stuff has paid off so well for me in the past. There's a bunch of code in the ParallelMesh synchronization, in one of our apps, and even a bit in Parallel:: itself that I think I could greatly simplify with a little of this refactoring. --- Roy |
From: Cody P. <cod...@gm...> - 2013-04-05 17:04:47
|
Alright gents, I'm finally getting back to this task. We kinda need this now since we are running large jobs quite regularly and reading the mesh on every processor is not making our I/O system happy. I know that Derek had just stopped using "boundary names" so that we could just revert back to the parallel element broadcast routines, but we'd like to just get this all going for the benefit of our end users. I thought I'd warm up with just broadcasting a std::vector<std::string> since it doesn't appear that we can even do that at this point. From there it should be relatively easy to get a std::pair<T1, T2> where T1 or T2 is a string moved across the wire. I'm trying to wrap my head around how you guys intend for this to work. Do we implement the pack/unpack routines for std::basic_string<T> and use those in conjunction with the broadcast or do we just create additional specializations for certain Parallel routines directly? I haven't played around at this layer long enough to know what you might be thinking. Perhaps if you can provide a few lines of sample code on how you see the API working might help me get started as well. Thanks, Cody On Wed, Feb 13, 2013 at 2:49 PM, Roy Stogner <roy...@ic...>wrote: > > On Wed, 13 Feb 2013, Derek Gaston wrote: > > > How about just finding the max length for the names and reading all > > the names into fixed length char*s.... and just using the available > > routines to send them across the wire. > > That would definitely work. > > > We don't need to over think this... we are sending some strings > > across the wire once per simulation...... > > Once per repartitioning, at most. > > But yeah, I'm not overthinking this because infrequently synchronizing > a few strings is still hard; rather because taking time to overthink > Parallel:: stuff has paid off so well for me in the past. There's a > bunch of code in the ParallelMesh synchronization, in one of our apps, > and even a bit in Parallel:: itself that I think I could greatly > simplify with a little of this refactoring. > --- > Roy > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > Libmesh-devel mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-devel > |