libMesh: A C++ Finite Element Library

On Mon, 28 Apr 2008, Martin Lüthi wrote:

> After being able to compile SVN-Libmesh with the option
>
> ./configure --with-cxx=mpicxx --with-cc=mpicc --with-f77=mpif77
>
> together with PETSC petsc-2.3.2-p10 (openmpi), I get the following
> error message when running example 13 (it *has to be* the 13!):
>
> *** Solving time step 0, time = 0.01 ***
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Object is in wrong state!
> [0]PETSC ERROR:  Scatter ctx already in use!
> [0]PETSC ERROR: ------------------------------------------------------------------------

> [0]PETSC ERROR: VecScatterBegin() line 1494 in src/vec/vec/utils/vscat.c
> [0]PETSC ERROR: User provided function() line 596 in unknowndirectory/src/numerics/petsc_vector.C
> [yak:12973] MPI_ABORT invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0 with errorcode 73
> make: *** [run] Error 73
>
>
> any ideas how to proceed from here? `make echo` is attached.

The first place to start is to recompile and run in debug mode;
hopefully the error will be repeatable there, but the failure will be
at an assertion in libMesh code rather than something in PETSc.
---
Roy

Hi,

I'm not sure how much testing we ever did on PETSc 2.3.2-p10...
Unless you are stuck with that PETSc version you might try whatever
the latest 2.3.3 series release is, and see if it has the same issue.

However, like Roy mentioned, I am also curious to see if any asserts
are thrown in debug mode and I'd like to fix any bugs we might have
with that PETSc version on our end.

-J

On Mon, Apr 28, 2008 at 7:42 AM, Martin Lüthi <luethi@...> wrote:
> Hi there
>
>  After being able to compile SVN-Libmesh with the option
>
>  ./configure --with-cxx=mpicxx --with-cc=mpicc --with-f77=mpif77
>
>  together with PETSC petsc-2.3.2-p10 (openmpi), I get the following
>  error message when running example 13 (it *has to be* the 13!):
>
>  *** Solving time step 0, time = 0.01 ***
>  [0]PETSC ERROR: --------------------- Error Message ------------------------------------
>  [0]PETSC ERROR: Object is in wrong state!
>  [0]PETSC ERROR:  Scatter ctx already in use!
>  [0]PETSC ERROR: ------------------------------------------------------------------------
>  [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 3, Fri Jun 15 16:51:25 CDT 2007 HG revision: f051789beadcd36f77fb6111d20225e26ed7cc0d
>  [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>  [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>  [0]PETSC ERROR: See docs/index.html for manual pages.
>  [0]PETSC ERROR: ------------------------------------------------------------------------
>  [0]PETSC ERROR: ./ex13-opt on a linux-gnu named yak by tinu Mon Apr 28 12:36:12 2008
>  [0]PETSC ERROR: Libraries linked from /build/buildd/petsc-2.3.3/lib/linux-gnu-c-opt
>  [0]PETSC ERROR: Configure run at Mon Mar 17 08:50:12 2008
>  [0]PETSC ERROR: Configure options --with-shared --with-dynamic --with-debugging=0 --useThreads 0 --with-mpi-dir=/usr --with-mpi-shared=1 --with-blas-lib=-lblas --with-lapack-lib=-llapack --with-umfpack=1 --with-umfpack-include=/usr/include/suitesparse --with-umfpack-lib="[/usr/lib/libumfpack.so,/usr/lib/libamd.so]" --with-superlu=1 --with-superlu-include=/usr/include/superlu --with-superlu-lib=/usr/lib/libsuperlu.so
>  [0]PETSC ERROR: ------------------------------------------------------------------------
>  [0]PETSC ERROR: VecScatterBegin() line 1494 in src/vec/vec/utils/vscat.c
>  [0]PETSC ERROR: User provided function() line 596 in unknowndirectory/src/numerics/petsc_vector.C
>  [yak:12973] MPI_ABORT invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0 with errorcode 73
>  make: *** [run] Error 73
>
>
>  any ideas how to proceed from here? `make echo` is attached.
>
>  Thanks
>
>  Martin
>
>
> -------------------------------------------------------------------------
>  This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>  Don't miss this year's exciting event. There's still time to save $100.
>  Use priority code J8TL2D2.
>  http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
>  Libmesh-devel mailing list
>  Libmesh-devel@...
>  https://lists.sourceforge.net/lists/listinfo/libmesh-devel
>
>

Hi

Thanks for your answers. While I could not figure out what was wrong
(I got no more information in debug mode), using PETSc 2.3.3-12 works
smoothly. The crucial thing seems to be the usage of the mpi
compilers. 

>  ./configure --with-cxx=mpicxx --with-cc=mpicc --with-f77=mpif77

Should that be documented on the "Install" page?

Best, Martin

On Mon, 28 Apr 2008, Martin Lüthi wrote:

> Thanks for your answers. While I could not figure out what was wrong
> (I got no more information in debug mode), using PETSc 2.3.3-12 works
> smoothly. The crucial thing seems to be the usage of the mpi
> compilers.
>
>>  ./configure --with-cxx=mpicxx --with-cc=mpicc --with-f77=mpif77
>
> Should that be documented on the "Install" page?

Actually, it probably ought to be fixed in our ./configure scripts.
We shouldn't be forcing users to specify mpi compiler scripts
manually, but we're using the default AC_PROG_CC and AC_PROG_CXX
macros to search for compilers, and (without arguments) those don't
search for mpicc or mpicxx.  I don't even see where we're getting the
--with-f77 option; some other macro must be calling AC_PROG_F77 I
suppose.

Does anyone more familiar than I am with autoconf want to take a look
at this?  Using something like:
AC_PROG_CC([mpicc gcc cc])
(along with equivalent CXX, F77 lines) looks like a quick fix, but
we'd have to move around some tests, I think, since we'd rather not
test for mpicc unless MPI is enabled.

It'll be a while before I can get to this myself; my source tree is
currently a bit out of sync due to ParallelMesh code changes, and I'd
rather not futz with something else before I get around to finishing
and merging those.
---
Roy

Why is this an issue though?  I've never specified --with-cxx or
--with-cc with libMesh before... and never had a problem because I
specify MPI_HOME correctly...

Also... around Sandia we don't use mpicc or mpicxx... as they are
sometimes buggy (especially on certain supercomputers with very
specific mpi setups)... we prefer to interrogate the mpi setup and add
the library and include paths ourselves...

Is there something I'm missing?  I guess I just prefer to call g++ directly...

Derek

On Mon, Apr 28, 2008 at 3:22 PM, Roy Stogner <roy@...> wrote:
>
>  On Mon, 28 Apr 2008, Martin Lüthi wrote:
>
>
> > Thanks for your answers. While I could not figure out what was wrong
> > (I got no more information in debug mode), using PETSc 2.3.3-12 works
> > smoothly. The crucial thing seems to be the usage of the mpi
> > compilers.
> >
> >
> > >  ./configure --with-cxx=mpicxx --with-cc=mpicc --with-f77=mpif77
> > >
> >
> > Should that be documented on the "Install" page?
> >
>
>  Actually, it probably ought to be fixed in our ./configure scripts.
>  We shouldn't be forcing users to specify mpi compiler scripts
>  manually, but we're using the default AC_PROG_CC and AC_PROG_CXX
>  macros to search for compilers, and (without arguments) those don't
>  search for mpicc or mpicxx.  I don't even see where we're getting the
>  --with-f77 option; some other macro must be calling AC_PROG_F77 I
>  suppose.
>
>  Does anyone more familiar than I am with autoconf want to take a look
>  at this?  Using something like:
>  AC_PROG_CC([mpicc gcc cc])
>  (along with equivalent CXX, F77 lines) looks like a quick fix, but
>  we'd have to move around some tests, I think, since we'd rather not
>  test for mpicc unless MPI is enabled.
>
>  It'll be a while before I can get to this myself; my source tree is
>  currently a bit out of sync due to ParallelMesh code changes, and I'd
>  rather not futz with something else before I get around to finishing
>  and merging those.
>  ---
>  Roy
> -------------------------------------------------------------------------
>  This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>  Don't miss this year's exciting event. There's still time to save $100.
>  Use priority code J8TL2D2.
>
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
>  Libmesh-devel mailing list
>  Libmesh-devel@...
>  https://lists.sourceforge.net/lists/listinfo/libmesh-devel
>
>

It pretty much depends on the PETSc install.  If PETSc is configured using --with-mpi-dir or something similar then libMesh will pick that up from the petscconf file.  If PETSc is configured to use mpicc etc... directly, then
libMesh can't snoop PETSc's MPI config, and you better just use mpicxx etc...  for libMesh.

And F77 comes in indirectly...  We need to be able to link to Fortran libraries, which means autoconf needs to figure out what libraries are needed to link with Fortran-compiled objects with a C main function.  This requires access to the Fortran compiler.

The real issue is that the user needs to be sure to build libMesh with the same MPI PETSc (& in the future Trilinos) used.  To be safe, the same compiler too.  I've had problems in the past with a gcc libMesh on top of an intel-compiled PETSc.  (Note that both gcc and intel-compiler libMesh worked on top of a gcc-compiled PETSc.)

-Ben

 
 
________________________________

From: libmesh-devel-bounces@... on behalf of Derek Gaston
Sent: Mon 4/28/2008 8:08 PM
To: Roy Stogner
Cc: libmesh-devel@...
Subject: Re: [Libmesh-devel] Compilation / run / mpi problems



Why is this an issue though?  I've never specified --with-cxx or
--with-cc with libMesh before... and never had a problem because I
specify MPI_HOME correctly...

Also... around Sandia we don't use mpicc or mpicxx... as they are
sometimes buggy (especially on certain supercomputers with very
specific mpi setups)... we prefer to interrogate the mpi setup and add
the library and include paths ourselves...

Is there something I'm missing?  I guess I just prefer to call g++ directly...

Derek

On Mon, Apr 28, 2008 at 3:22 PM, Roy Stogner <roy@...> wrote:
>
>  On Mon, 28 Apr 2008, Martin Lüthi wrote:
>
>
> > Thanks for your answers. While I could not figure out what was wrong
> > (I got no more information in debug mode), using PETSc 2.3.3-12 works
> > smoothly. The crucial thing seems to be the usage of the mpi
> > compilers.
> >
> >
> > >  ./configure --with-cxx=mpicxx --with-cc=mpicc --with-f77=mpif77
> > >
> >
> > Should that be documented on the "Install" page?
> >
>
>  Actually, it probably ought to be fixed in our ./configure scripts.
>  We shouldn't be forcing users to specify mpi compiler scripts
>  manually, but we're using the default AC_PROG_CC and AC_PROG_CXX
>  macros to search for compilers, and (without arguments) those don't
>  search for mpicc or mpicxx.  I don't even see where we're getting the
>  --with-f77 option; some other macro must be calling AC_PROG_F77 I
>  suppose.
>
>  Does anyone more familiar than I am with autoconf want to take a look
>  at this?  Using something like:
>  AC_PROG_CC([mpicc gcc cc])
>  (along with equivalent CXX, F77 lines) looks like a quick fix, but
>  we'd have to move around some tests, I think, since we'd rather not
>  test for mpicc unless MPI is enabled.
>
>  It'll be a while before I can get to this myself; my source tree is
>  currently a bit out of sync due to ParallelMesh code changes, and I'd
>  rather not futz with something else before I get around to finishing
>  and merging those.
>  ---
>  Roy
> -------------------------------------------------------------------------
>  This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>  Don't miss this year's exciting event. There's still time to save $100.
>  Use priority code J8TL2D2.
>
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
>  Libmesh-devel mailing list
>  Libmesh-devel@...
>  https://lists.sourceforge.net/lists/listinfo/libmesh-devel
>
>

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
Libmesh-devel mailing list
Libmesh-devel@...
https://lists.sourceforge.net/lists/listinfo/libmesh-devel