libMesh: A C++ Finite Element Library

On Thu, 30 Aug 2007, David Knezevic wrote:

> if(libMesh::on_command_line("--force-sequential"))
> {
>   libMeshPrivateData::_processor_id=0;
>   libMeshPrivateData::_n_processors=1;
> }

How does this differ from the behavior of the "--disable-mpi" command
line argument?  (keeping in mind that I've never used that either...)
---
Roy

> How does this differ from the behavior of the "--disable-mpi" command
> line argument?  (keeping in mind that I've never used that either...)

Ahh they looks like a good idea! And I don't have to mess around with  
the source to try it out.

Thanks!
Dave

Petsc probably won't work with --disable-mpi on the command line.

-John



David Knezevic writes:
 > > How does this differ from the behavior of the "--disable-mpi" command
 > > line argument?  (keeping in mind that I've never used that either...)
 > 
 > Ahh they looks like a good idea! And I don't have to mess around with  
 > the source to try it out.
 > 
 > Thanks!
 > Dave

Yep you're right John. There's a libmesh_initialize_mpi flag which  
doesn't get set when using --disable-mpi.. I hit an assert or  
something, which says: "Attempting to use an MPI routine before  
intializeing MPICH".

Whereas making that change to libMesh.C I described in the previous  
email, PETSc works (at least on a single processor), since the  
libmesh_initialized_mpi flag still get set. Now to try it in parallel.

- dave



On 30/08/2007, at 10:48 PM, John Peterson wrote:

> Petsc probably won't work with --disable-mpi on the command line.
>
> -John
>
>
>
> David Knezevic writes:
>>> How does this differ from the behavior of the "--disable-mpi"  
>>> command
>>> line argument?  (keeping in mind that I've never used that  
>>> either...)
>>
>> Ahh they looks like a good idea! And I don't have to mess around with
>> the source to try it out.
>>
>> Thanks!
>> Dave

On Thu, 30 Aug 2007, John Peterson wrote:

> Petsc probably won't work with --disable-mpi on the command line.

Ah, good point.  We want MPI to be running, but with a different MPI
communicator on each process, is that right?  I've actually had that
happen by accident before when OpenMPI and MPICH had conflicting
binaries on the same machine, but I have no idea how you would go
about doing that on purpose... ;-)
---
Roy

> Ah, good point.  We want MPI to be running, but with a different MPI
> communicator on each process, is that right?  I've actually had that
> happen by accident before when OpenMPI and MPICH had conflicting
> binaries on the same machine, but I have no idea how you would go
> about doing that on purpose... ;-)

All that is needed is to initialize PETSc on each processor with  
PETSC_COMM_SELF, rather than PETSC_COMM_WORLD. This is exactly what  
is done when we use PETSc on a single processor, which is why I'm  
trying to fool libMesh into thinking it's running on a single processor.

- dave

Forgot to reply-all...



> Hi Ben,
>
>> What about
>>
>> libMesh::init (argc, argv, MPI_COMM_SELF);
>>
>> ?
>>
>> Check out src/base/libmesh.C...
>
> Thanks! That works! However, it ensures that an independent PETSc  
> is instantiated on each processor. It turns out that what I need is  
> a global PETSc instantiation (because I use this to partition the  
> PETSc dense matrix that I'm using to store the solution data).
>
> After a bit of trial and error I've got what I need to work. I've  
> used a --force-sequential flag to initialize libMesh's  
> MPI_Communicator (on line 146 libmesh.C) to MPI_COMM_SELF, but then  
> set PETSc's communicator to COMM_WORLD_IN on line 159...
>
> The only downside is that I get the performance data printed out  
> once for each processor, because within libMesh each processor  
> thinks that it is processor 0. How do you turn off the performance  
> data printout?
>
> Thanks,
> Dave
>
>
>
>
>
>
>> On 8/30/07 4:59 PM, "David Knezevic"  
>> <david.knezevic@...>
>> wrote:
>>
>>>> Ah, good point.  We want MPI to be running, but with a different  
>>>> MPI
>>>> communicator on each process, is that right?  I've actually had  
>>>> that
>>>> happen by accident before when OpenMPI and MPICH had conflicting
>>>> binaries on the same machine, but I have no idea how you would go
>>>> about doing that on purpose... ;-)
>>>
>>> All that is needed is to initialize PETSc on each processor with
>>> PETSC_COMM_SELF, rather than PETSC_COMM_WORLD. This is exactly what
>>> is done when we use PETSc on a single processor, which is why I'm
>>> trying to fool libMesh into thinking it's running on a single  
>>> processor.
>>>
>>> - dave
>>>
>>> -------------------------------------------------------------------- 
>>> -----
>>> This SF.net email is sponsored by: Splunk Inc.
>>> Still grepping through log files to find problems?  Stop.
>>> Now Search log events and configuration files using AJAX and a  
>>> browser.
>>> Download your FREE copy of Splunk now >>  http://get.splunk.com/
>>> _______________________________________________
>>> Libmesh-users mailing list
>>> Libmesh-users@...
>>> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>>
>

David Knezevic writes:
 > >
 > > The only downside is that I get the performance data printed out  
 > > once for each processor, because within libMesh each processor  
 > > thinks that it is processor 0. How do you turn off the performance  
 > > data printout?
During configure, --disable-perflog

-J

Look for the flag --keep-cout (src/base/libmesh.C, I think) -- that does the
inverse of what you want.

In parallel it nullifys std::cout for all processors except processor 0,
thus allowing you to use std::cout as normal and not get duplicate output.

So, somehow you need to use COMM_WORLD_IN to find out there are truly more
than one processor, attach std::cout to null on all but processor 0, and
then use COMM_SELF from there on out...


On 8/31/07 8:35 AM, "John Peterson" <peterson@...> wrote:

> David Knezevic writes:
>>> 
>>> The only downside is that I get the performance data printed out
>>> once for each processor, because within libMesh each processor
>>> thinks that it is processor 0. How do you turn off the performance
>>> data printout?
> During configure, --disable-perflog
> 
> -J
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> Libmesh-users mailing list
> Libmesh-users@...
> https://lists.sourceforge.net/lists/listinfo/libmesh-users

> Look for the flag --keep-cout (src/base/libmesh.C, I think) -- that  
> does the
> inverse of what you want.
>
> In parallel it nullifys std::cout for all processors except  
> processor 0,
> thus allowing you to use std::cout as normal and not get duplicate  
> output.
>
> So, somehow you need to use COMM_WORLD_IN to find out there are  
> truly more
> than one processor, attach std::cout to null on all but processor  
> 0, and
> then use COMM_SELF from there on out...

Ah, that's a very nice fix. I have the "real" processor ID available  
by calling MPI_Comm_rank(PETSC_COMM_WORLD, &proc_id), so just using  
that value in the "--keep-cout" test (rather than libMesh:processor_id 
() which I've artificially set to 0 on all processors) does the trick.

Thanks,
Dave


> On 8/31/07 8:35 AM, "John Peterson" <peterson@...>  
> wrote:
>
>> David Knezevic writes:
>>>>
>>>> The only downside is that I get the performance data printed out
>>>> once for each processor, because within libMesh each processor
>>>> thinks that it is processor 0. How do you turn off the performance
>>>> data printout?
>> During configure, --disable-perflog
>>
>> -J
>>
>> --------------------------------------------------------------------- 
>> ----
>> This SF.net email is sponsored by: Splunk Inc.
>> Still grepping through log files to find problems?  Stop.
>> Now Search log events and configuration files using AJAX and a  
>> browser.
>> Download your FREE copy of Splunk now >>  http://get.splunk.com/
>> _______________________________________________
>> Libmesh-users mailing list
>> Libmesh-users@...
>> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>