From: David K. <dav...@ba...> - 2007-08-30 21:34:32
|
Hi, Following up a couple of posts I sent a few days ago, I'm trying to get libMesh to do concurrent, independent sequential solves on a parallel machine. It seems to me that a simple way to do this would be to edit libMesh.C in the following way (around line 150): if(libMesh::on_command_line("--force-sequential")) { libMeshPrivateData::_processor_id=0; libMeshPrivateData::_n_processors=1; } else { // this is how _processor_id and _n_processors are currently set MPI_Comm_rank(libMesh::COMM_WORLD, ..... etc } Therefore if I passed this --force-sequential flag on the command line, libMesh would do a sequential solve, and the mesh wouldn't be partitioned or anything. I haven't been able to try this out yet on a parallel machine, but I was wondering if anyone sees any obvious problems with this idea? Cheers, Dave |
From: David K. <dav...@gm...> - 2007-08-31 12:50:30
|
Forgot to reply-all... > Hi Ben, > >> What about >> >> libMesh::init (argc, argv, MPI_COMM_SELF); >> >> ? >> >> Check out src/base/libmesh.C... > > Thanks! That works! However, it ensures that an independent PETSc > is instantiated on each processor. It turns out that what I need is > a global PETSc instantiation (because I use this to partition the > PETSc dense matrix that I'm using to store the solution data). > > After a bit of trial and error I've got what I need to work. I've > used a --force-sequential flag to initialize libMesh's > MPI_Communicator (on line 146 libmesh.C) to MPI_COMM_SELF, but then > set PETSc's communicator to COMM_WORLD_IN on line 159... > > The only downside is that I get the performance data printed out > once for each processor, because within libMesh each processor > thinks that it is processor 0. How do you turn off the performance > data printout? > > Thanks, > Dave > > > > > > >> On 8/30/07 4:59 PM, "David Knezevic" >> <dav...@ba...> >> wrote: >> >>>> Ah, good point. We want MPI to be running, but with a different >>>> MPI >>>> communicator on each process, is that right? I've actually had >>>> that >>>> happen by accident before when OpenMPI and MPICH had conflicting >>>> binaries on the same machine, but I have no idea how you would go >>>> about doing that on purpose... ;-) >>> >>> All that is needed is to initialize PETSc on each processor with >>> PETSC_COMM_SELF, rather than PETSC_COMM_WORLD. This is exactly what >>> is done when we use PETSc on a single processor, which is why I'm >>> trying to fool libMesh into thinking it's running on a single >>> processor. >>> >>> - dave >>> >>> -------------------------------------------------------------------- >>> ----- >>> This SF.net email is sponsored by: Splunk Inc. >>> Still grepping through log files to find problems? Stop. >>> Now Search log events and configuration files using AJAX and a >>> browser. >>> Download your FREE copy of Splunk now >> http://get.splunk.com/ >>> _______________________________________________ >>> Libmesh-users mailing list >>> Lib...@li... >>> https://lists.sourceforge.net/lists/listinfo/libmesh-users >> > |
From: John P. <pet...@cf...> - 2007-08-31 13:35:33
|
David Knezevic writes: > > > > The only downside is that I get the performance data printed out > > once for each processor, because within libMesh each processor > > thinks that it is processor 0. How do you turn off the performance > > data printout? During configure, --disable-perflog -J |
From: Benjamin K. <ben...@na...> - 2007-08-31 13:50:54
|
Look for the flag --keep-cout (src/base/libmesh.C, I think) -- that does the inverse of what you want. In parallel it nullifys std::cout for all processors except processor 0, thus allowing you to use std::cout as normal and not get duplicate output. So, somehow you need to use COMM_WORLD_IN to find out there are truly more than one processor, attach std::cout to null on all but processor 0, and then use COMM_SELF from there on out... On 8/31/07 8:35 AM, "John Peterson" <pet...@cf...> wrote: > David Knezevic writes: >>> >>> The only downside is that I get the performance data printed out >>> once for each processor, because within libMesh each processor >>> thinks that it is processor 0. How do you turn off the performance >>> data printout? > During configure, --disable-perflog > > -J > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Libmesh-users mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-users |
From: David K. <dav...@gm...> - 2007-08-31 16:56:29
|
> Look for the flag --keep-cout (src/base/libmesh.C, I think) -- that > does the > inverse of what you want. > > In parallel it nullifys std::cout for all processors except > processor 0, > thus allowing you to use std::cout as normal and not get duplicate > output. > > So, somehow you need to use COMM_WORLD_IN to find out there are > truly more > than one processor, attach std::cout to null on all but processor > 0, and > then use COMM_SELF from there on out... Ah, that's a very nice fix. I have the "real" processor ID available by calling MPI_Comm_rank(PETSC_COMM_WORLD, &proc_id), so just using that value in the "--keep-cout" test (rather than libMesh:processor_id () which I've artificially set to 0 on all processors) does the trick. Thanks, Dave > On 8/31/07 8:35 AM, "John Peterson" <pet...@cf...> > wrote: > >> David Knezevic writes: >>>> >>>> The only downside is that I get the performance data printed out >>>> once for each processor, because within libMesh each processor >>>> thinks that it is processor 0. How do you turn off the performance >>>> data printout? >> During configure, --disable-perflog >> >> -J >> >> --------------------------------------------------------------------- >> ---- >> This SF.net email is sponsored by: Splunk Inc. >> Still grepping through log files to find problems? Stop. >> Now Search log events and configuration files using AJAX and a >> browser. >> Download your FREE copy of Splunk now >> http://get.splunk.com/ >> _______________________________________________ >> Libmesh-users mailing list >> Lib...@li... >> https://lists.sourceforge.net/lists/listinfo/libmesh-users > |
From: Roy S. <roy...@ic...> - 2007-08-30 21:40:32
|
On Thu, 30 Aug 2007, David Knezevic wrote: > if(libMesh::on_command_line("--force-sequential")) > { > libMeshPrivateData::_processor_id=0; > libMeshPrivateData::_n_processors=1; > } How does this differ from the behavior of the "--disable-mpi" command line argument? (keeping in mind that I've never used that either...) --- Roy |
From: David K. <dav...@ba...> - 2007-08-30 21:43:49
|
> How does this differ from the behavior of the "--disable-mpi" command > line argument? (keeping in mind that I've never used that either...) Ahh they looks like a good idea! And I don't have to mess around with the source to try it out. Thanks! Dave |
From: John P. <pet...@cf...> - 2007-08-30 21:49:06
|
Petsc probably won't work with --disable-mpi on the command line. -John David Knezevic writes: > > How does this differ from the behavior of the "--disable-mpi" command > > line argument? (keeping in mind that I've never used that either...) > > Ahh they looks like a good idea! And I don't have to mess around with > the source to try it out. > > Thanks! > Dave |
From: David K. <dav...@ba...> - 2007-08-30 21:56:16
|
Yep you're right John. There's a libmesh_initialize_mpi flag which doesn't get set when using --disable-mpi.. I hit an assert or something, which says: "Attempting to use an MPI routine before intializeing MPICH". Whereas making that change to libMesh.C I described in the previous email, PETSc works (at least on a single processor), since the libmesh_initialized_mpi flag still get set. Now to try it in parallel. - dave On 30/08/2007, at 10:48 PM, John Peterson wrote: > Petsc probably won't work with --disable-mpi on the command line. > > -John > > > > David Knezevic writes: >>> How does this differ from the behavior of the "--disable-mpi" >>> command >>> line argument? (keeping in mind that I've never used that >>> either...) >> >> Ahh they looks like a good idea! And I don't have to mess around with >> the source to try it out. >> >> Thanks! >> Dave |
From: Roy S. <roy...@ic...> - 2007-08-30 21:57:11
|
On Thu, 30 Aug 2007, John Peterson wrote: > Petsc probably won't work with --disable-mpi on the command line. Ah, good point. We want MPI to be running, but with a different MPI communicator on each process, is that right? I've actually had that happen by accident before when OpenMPI and MPICH had conflicting binaries on the same machine, but I have no idea how you would go about doing that on purpose... ;-) --- Roy |
From: David K. <dav...@ba...> - 2007-08-30 21:59:48
|
> Ah, good point. We want MPI to be running, but with a different MPI > communicator on each process, is that right? I've actually had that > happen by accident before when OpenMPI and MPICH had conflicting > binaries on the same machine, but I have no idea how you would go > about doing that on purpose... ;-) All that is needed is to initialize PETSc on each processor with PETSC_COMM_SELF, rather than PETSC_COMM_WORLD. This is exactly what is done when we use PETSc on a single processor, which is why I'm trying to fool libMesh into thinking it's running on a single processor. - dave |