Menu ▾ ▴

Tutorial seg faulting

Help
Scott H
2014-07-18
2014-07-19

  • Scott H

    Scott H - 2014-07-18

    Hi, my student and I are running two different Macs (mine Mavericks and his Mountain Lion) and did not (knowingly) link against Apple's vecLib, yet we're both still getting segmentation faults in the same place when we try to run the first tutorial.

    We compiled OpenMPI from source as directed, used MacPorts wherever not specified (using package "libconfig-hr" for a recent version of libconfig), and configured AcouSTO via...

    % ./configure --with-mpicc=/usr/local/bin/mpicc --enable-mpinew --with-libconfig=/opt/local/lib --with-blacs=$HOME/apps/scalapack_installer_1.0.2/install/lib --with-scalapack=$HOME/apps/scalapack_installer_1.0.2/install/lib --enable-scalapack_new --with-lapack=$HOME/apps/scalapack_installer_1.0.2/install/lib --with-blas=$HOME/apps/scalapack_installer_1.0.2/install/lib

    When running, with or without MPI, we see the following...

    % acousto -f sphere_source.cfg
2014-07-18 16:26:11 INFO  [1/1] ****************    Job started     ***************
2014-07-18 16:26:11 INFO  [1/1] Program started
2014-07-18 16:26:11 INFO  [1/1] Creating directory /private/tmp/acousto/pro-ch0-pc1-gmr-2014-07-18
2014-07-18 16:26:11 WARN  [1/1] WARNING! Directory pro-ch0-pc1-gmr-2014-07-18 exists, existing files will be OVERWRITTEN
2014-07-18 16:26:11 INFO  [1/1] Warning: Grid size was automatically determined as 1 x 1
2014-07-18 16:26:11 INFO  [1/1]          This choice is not optimal for large problems and a high number of processes.
2014-07-18 16:26:11 INFO  [1/1]          It may also lead to distribution inconsistencies. 
2014-07-18 16:26:11 INFO  [1/1] Mics file appears to be in AcouSTO native format
2014-07-18 16:26:11 INFO  [1/1]     Number of mics is read from config file
2014-07-18 16:26:11 INFO  [1/1] n geometries=1
2014-07-18 16:26:11 INFO  [1/1] GMRES Solver chosen
2014-07-18 16:26:11 INFO  [1/1]  Initializing process grid [1 x 1]
2014-07-18 16:26:11 INFO  [1/1]  Successfully initialized process grid [1 x 1]
2014-07-18 16:26:11 INFO  [1/1] ****************    Memory Estimate    ***************
2014-07-18 16:26:11 INFO  [1/1] Consider this as a rough estimate. 
2014-07-18 16:26:11 INFO  [1/1] Memory actually allocated heavily depends on 
2014-07-18 16:26:11 INFO  [1/1]  the particular combination of parameters used  
2014-07-18 16:26:11 INFO  [1/1] Estimated Memory for Each Process: 53.551088 Mb
2014-07-18 16:26:11 INFO  [1/1] ****************    Memory Estimate-end    ***********
2014-07-18 16:26:11 INFO  [1/1] Starting Geometry     Module
2014-07-18 16:26:11 INFO  [1/1] Starting Coefficients Module
2014-07-18 16:26:11 INFO  [1/1] Starting Body Coefficients Module
2014-07-18 16:26:11 INFO  [1/1] Completed 10 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 20 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 30 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 40 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 50 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 60 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 70 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 80 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 90 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 100 % 
2014-07-18 16:26:11 INFO  [1/1] Body Coefficients Module End
2014-07-18 16:26:11 INFO  [1/1] Starting Mics Coefficients Module
2014-07-18 16:26:11 INFO  [1/1] Completed 10 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 20 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 30 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 40 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 50 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 60 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 70 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 80 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 90 % 
2014-07-18 16:26:11 INFO  [1/1] Completed 100 % 
2014-07-18 16:26:11 INFO  [1/1] Mics Coefficients Module End
2014-07-18 16:26:11 INFO  [1/1] Starting Solution     Module
2014-07-18 16:26:11 INFO  [1/1] ifreq = 0, s=(0.000000 +i 1077.566280), f=171.500000 Hz
2014-07-18 16:26:11 INFO  [1/1] Boundary conditions: transferring 1176 bytes to nodes
2014-07-18 16:26:11 INFO  [1/1] Starting GMRES iterations: maxit=1, restart=24
[provision:41283] *** Process received signal ***
[provision:41283] Signal: Segmentation fault: 11 (11)
[provision:41283] Signal code:  (0)
[provision:41283] Failing at address: 0x0
[provision:41283] [ 0] 0   libsystem_platform.dylib            0x00007fff966bf5aa _sigtramp + 26
[provision:41283] [ 1] 0   libsystem_malloc.dylib              0x00007fff9632a868 malloc_zone_malloc + 71
[provision:41283] [ 2] 0   acousto                             0x0000000103120b52 zvvdotc_ + 50
[provision:41283] [ 3] 0   acousto                             0x00000001030fa823 PB_CpdotNN + 1123
[provision:41283] [ 4] 0   acousto                             0x000000010312de44 pzdotc_ + 1188
[provision:41283] [ 5] 0   acousto                             0x00000001030ccae4 gmres + 3396
[provision:41283] [ 6] 0   acousto                             0x00000001030cb42b acoustic_solution + 3803
[provision:41283] [ 7] 0   acousto                             0x00000001030c2ba1 main + 21697
[provision:41283] [ 8] 0   libdyld.dylib                       0x00007fff8b5e25fd start + 1
[provision:41283] [ 9] 0   ???                                 0x0000000000000003 0x0 + 3
[provision:41283] *** End of error message ***
Segmentation fault

    Can someone help us understand this and hopefully fix it?

    Thanks.

     

  • Umberto Iemma

    Umberto Iemma - 2014-07-19

    You are using the pure tutorial without any change of the configuration files, right? Does this happen with all the tutorials?

     

  • Umberto Iemma

    Umberto Iemma - 2014-07-19

    It is clearly a blas problem. I won't have a mac at hand for all the w.e., so you have to wait till Monday. In the meanwhile, could you please provide more info about your environment.
    System version, ompi version, macport version etc. etc.
    I'm also running both Mavericks and MLion and I've never had this problem.
    Could you also check that acousto is linking exactly against the same blas/lapack as scalapack .

     

  • Umberto Iemma

    Umberto Iemma - 2014-07-19

    It is clearly a blas problem. I won't have a mac at hand for all the w.e., so you have to wait till Monday. In the meanwhile, could you please provide more info about your environment.
    System version, ompi version, macport version etc. etc.
    I'm also running both Mavericks and MLion and I've never had this problem.
    Could you also check that acousto is linking exactly against the same blas/lapack as scalapack .

     

  • Scott H

    Scott H - 2014-07-19

    UPDATE: No need to read this. Fixed it. See two-posts below.

    Thanks for your reply umberto.

    Yes, the config file was not mofied.

    I'm using OS X 10.9.4. And my student had 10.7.5. We both used OpenMPI 1.8.1, and the latest updates from MacPorts, scalapack installer 1.0.2, which downloaded scalapack 2.02.
    We were following instructions for installing on OS X 10.7.5 at 

    http://acousto.sourceforge.net/user_manual/html/UserManualse31.html

    (I only later found the instructions for Mavericks, but didn't want to switch from MacPorts to Homebrew since I use MacPorts for other applications.)

    For linking, we used the final output instructions from the scalapack installer to specify blas & lapack directories, i.e. where the installer said "Your BLAS library is -L...etc" (took off the -L for the Acousto config, as shown in the OP).

    I notice there are lapack and blas libs in /usr/lib, but since we specified for AcouSTO to use scalapack's versions in the configure script, I'm not sure how these might interfere.

    Here's the full output from AcouSTO's configure:

    % ./configure --with-mpicc=/usr/local/bin/mpicc --enable-mpinew --with-libconfig=/opt/local/lib --with-blacs=$HOME/apps/scalapack_installer_1.0.2/install/lib --with-scalapack=$HOME/apps/scalapack_installer_1.0.2/install/lib --enable-scalapack_new --with-lapack=$HOME/apps/scalapack_installer_1.0.2/install/lib --with-blas=$HOME/apps/scalapack_installer_1.0.2/install/lib
checking for a BSD-compatible install... usage: install installation-prefix

Example: install /usr/local

Note: See script source for user-modifiable environment variables that may 
need to be tailored to your desired compilation environment.
/usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... /opt/local/bin/gcc-mp-4.4
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /opt/local/bin/gcc-mp-4.4 accepts -g... yes
checking for /opt/local/bin/gcc-mp-4.4 option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of /opt/local/bin/gcc-mp-4.4... gcc3
checking whether /opt/local/bin/gcc-mp-4.4 and cc understand -c and -o together... yes
checking whether make sets $(MAKE)... (cached) yes
checking for robodoc... false
checking build system type... i686-apple-darwin13.3.0
checking host system type... i686-apple-darwin13.3.0
checking for config_init in -lconfig... yes
checking for dgemm_ in -lblas... yes
checking for zgegv_ in -llapack... yes
configure: MPI version forced to 1.7.x
configure: MPI version used: 1.7.x
checking for sl_init_ in -lscalapack... yes
configure: --------------------------
configure: -------- Summary ---------
configure: --------------------------
configure: BLAS     : -lblas
configure: ATLAS    : 
configure: LAPACK   : -llapack
configure: BLACS    : 
configure: SCALAPACK: -lscalapack
configure: LDFLAGS  : -L/opt/local/lib  -L/Users/shawley/apps/scalapack_installer_1.0.2/install/lib -L/Users/shawley/apps/scalapack_installer_1.0.2/install/lib -L/Users/shawley/apps/scalapack_installer_1.0.2/install/lib
configure: CFLAGS   : -g -O2
configure: FORTRAN  : -lgfortran
configure: CFLAGS   : -g -O2
configure: LDFLAGS  : -L/opt/local/lib  -L/Users/shawley/apps/scalapack_installer_1.0.2/install/lib -L/Users/shawley/apps/scalapack_installer_1.0.2/install/lib -L/Users/shawley/apps/scalapack_installer_1.0.2/install/lib
configure: MPI_VER  : 1.7.x
configure: --------------------------
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: executing depfiles commands
%

    One thing I notice: it says the MPI version is set to 1.7, but I specified which mpicc to use, and that was built via OpenMPI version 1.8.1. ?? There is no other mpicc on my system.

     

    Last edit: Scott H 2014-07-19

  • Scott H

    Scott H - 2014-07-19

    edit: ignore/delete this, see next reply.

     

    Last edit: Scott H 2014-07-19

  • Scott H

    Scott H - 2014-07-19

    Found and fixed the problem: 

    % ls ~/apps/scalapack_installer_1.0.2/install/lib
librefblas.a    libreflapack.a  libscalapack.a  libtmg.a

    There is actually no libblas.a or liblapack.a created by the scalapack installer!
    So all the linker could link to was Apple's versions, /usr/lib/libblas.a, etc.
    (Why would the scalapack people do that??)

    Fixed it by adding symbolic links, i.e.

    cd ~/apps/scalapack_installer_1.0.2/install/lib
ln -s librefblas.a libblas.a
ln -s libreflapack.a liblapack.a

    Rebuilt AcouSTO. Now the tutorial runs to completion. Thanks.

     

Log in to post a comment.