Ben et. al.....

I'm having some crazy issue with getting libMesh executables to link on a BlueGene/Q (Mira at Argonne).  The issue is that libtool is choosing to try to link dynamic libraries even though I explicitly set --enable-static --disable-shared.  It's weird because libtool is picking up .so's and putting them right on the link line instead of letting the linker find the static libraries that are sitting right next to those .so's.

I can manually create link lines that work - and generate working binaries... but I can't get libtool to do it automatically...

Below is a full description of what is happening... any help I could get would be awesome...

Derek

---------- Forwarded message ----------
From: Gaston, Derek R <derek.gaston@inl.gov>
Date: Thu, Aug 1, 2013 at 12:45 PM
Subject: libtool Trouble on Mira
To: Jeff Hammond <jhammond@alcf.anl.gov>, "John Peterson (INL)" <JW.Peterson@inl.gov>


Jeff,

(Note: If there is someone else I should send this to instead let me know)

I've been able to log into Mira and have been working on getting everything compiled.  So far, everything is going well.  I've been able to compile Hypre and PETSc without issue and I can _compile_ libMesh... but something is going wrong in the linking stage when trying to link an executable with libMesh.

What's happening is that libtool is picking up shared libraries for system libraries such as libmpichf90.  I cannot see why this is happening.  If I manually edit the link line to just link -lmpichf90 (and others) everything works fine and my executables are fine.

I am using the +mpiwrappers-gcc softenv.

Here's a typical libtool line that the libMesh build system is trying to run:

libtool --tag=CXX   --mode=link mpicxx -O0 -felide-constructors -g -ansi -pedantic -W -Wall -Wextra -Wno-long-long -Wunused -Wpointer-arith -Wformat -Wparentheses -std=c++0x -Woverloaded-virtual   -fopenmp    -o meshtool-dbg src/apps/meshtool_dbg-meshtool.o libmesh_dbg.la -L/home/gastdr/moose-compilers/petsc/petsc-3.3-p4/gcc-opt/lib -lpetsc -Wl,-rpath,/home/gastdr/moose-compilers/hypre/hypre-2.8.0b/gcc-opt/lib -L/home/gastdr/moose-compilers/hypre/hypre-2.8.0b/gcc-opt/lib -lHYPRE -L/bgsys/drivers/V1R2M0/ppc64/spi/lib -L/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/lib/gcc -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib -Wl,-rpath,/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -Wl,-rpath,/home/gastdr/moose-compilers/petsc/petsc-3.3-p4/gcc-opt/lib -lflapack -lfblas -lmpichf90 -lgfortran -lm -lcxxmpich -lmpich -lopa -lmpl -lpami -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lgomp -lgcc_eh -lnss_files -lnss_dns -lresolv -ldl


And here is the actual link line that libtool is creating:

mpicxx -O0 -felide-constructors -g -ansi -pedantic -W -Wall -Wextra -Wno-long-long -Wunused -Wpointer-arith -Wformat -Wparentheses -std=c++0x -Woverloaded-virtual -fopenmp -o meshtool-dbg src/apps/meshtool_dbg-meshtool.o -Wl,-rpath -Wl,/home/gastdr/moose-compilers/hypre/hypre-2.8.0b/gcc-opt/lib -Wl,-rpath -Wl,/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -Wl,-rpath -Wl,/home/gastdr/moose-compilers/petsc/petsc-3.3-p4/gcc-opt/lib  ./.libs/libmesh_dbg.a -L./src/mpl -L./src/openpa/src -L/home/gastdr/moose-compilers/petsc/petsc-3.3-p4/gcc-opt/lib -L/home/gastdr/moose-compilers/hypre/hypre-2.8.0b/gcc-opt/lib -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib /gpfs/mira-home/gastdr/projects/libmesh/build/contrib/netcdf/v4/liblib/.libs/libnetcdf.a -lpetsc -lHYPRE -L/bgsys/drivers/V1R2M0/ppc64/spi/lib -L/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/lib/gcc -lflapack -lfblas /bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib/libmpichf90.so /bgsys/drivers/toolchain/V1R2M0/gnu-linux/powerpc64-bgq-linux/lib/libgfortran.so /bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib/libgfortran.so /bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib/libcxxmpich.so /bgsys/drivers/toolchain/V1R2M0/gnu-linux/powerpc64-bgq-linux/lib/libstdc++.so /bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib/libmpich.so /bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib/libopa.so /bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib/libmpl.so -lpami -lSPI -lSPI_cnk -lpthread /bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib/libstdc++.so -lm /bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib/libgomp.so -lrt -lgcc_eh -lnss_files -lnss_dns -lresolv -ldl -fopenmp -Wl,-rpath -Wl,/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib -Wl,-rpath -Wl,/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -Wl,-rpath -Wl,/bgsys/drivers/toolchain/V1R2M0/gnu-linux/powerpc64-bgq-linux/lib -Wl,-rpath -Wl,/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib -Wl,-rpath -Wl,/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -Wl,-rpath -Wl,/bgsys/drivers/toolchain/V1R2M0/gnu-linux/powerpc64-bgq-linux/lib


Note the .so files that are directly on the link line.  This gives rise to the following error:

/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6/../../../../powerpc64-bgq-linux/bin/ld: attempted static link of dynamic object `/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib/libmpichf90.so'


If I remove all of the .so's from that link line (and add -lgfortran) I get a link line that works:

mpicxx -O0 -felide-constructors -g -ansi -pedantic -W -Wall -Wextra -Wno-long-long -Wunused -Wpointer-arith -Wformat -Wparentheses -std=c++0x -Woverloaded-virtual -fopenmp -o meshtool-dbg src/apps/meshtool_dbg-meshtool.o -Wl,-rpath -Wl,/home/gastdr/moose-compilers/hypre/hypre-2.8.0b/gcc-opt/lib -Wl,-rpath -Wl,/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -Wl,-rpath -Wl,/home/gastdr/moose-compilers/petsc/petsc-3.3-p4/gcc-opt/lib  ./.libs/libmesh_dbg.a -L./src/mpl -L./src/openpa/src -L/home/gastdr/moose-compilers/petsc/petsc-3.3-p4/gcc-opt/lib -L/home/gastdr/moose-compilers/hypre/hypre-2.8.0b/gcc-opt/lib -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib /gpfs/mira-home/gastdr/projects/libmesh/build/contrib/netcdf/v4/liblib/.libs/libnetcdf.a -lpetsc -lHYPRE -L/bgsys/drivers/V1R2M0/ppc64/spi/lib -L/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/lib/gcc -lflapack -lfblas -lpami -lSPI -lSPI_cnk -lpthread  -lm  -lrt -lgcc_eh -lnss_files -lnss_dns -lresolv -ldl -fopenmp -Wl,-rpath -Wl,/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib -Wl,-rpath -Wl,/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -Wl,-rpath -Wl,/bgsys/drivers/toolchain/V1R2M0/gnu-linux/powerpc64-bgq-linux/lib -Wl,-rpath -Wl,/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib -Wl,-rpath -Wl,/bgsys/drivers/V1R2M0/ppc64/comm/gcc/lib -Wl,-rpath -Wl,/bgsys/drivers/toolchain/V1R2M0/gnu-linux/powerpc64-bgq-linux/lib -lgfortran


I went and looked at the .la files for those libraries (for instance /bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib/libgfortran.la) and they definitely are specifying the "old_library" option like they are supposed to for the static version of the library.

I'm really stumped on this one...

Derek