#64 Dynamic linkage to libgfortran and libquadmath with JAGS 5.0.0

[Anonymous]

A number of the binary files for JAGS 5.0.0 are built with dynamic linkage to gfortran and quadmath, for example otool -L gives:

src/lib/.libs/libjags.5.dylib:
/usr/local/lib/libjags.5.dylib (compatibility version 6.0.0, current version 6.0.0)
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1356.0.0)
/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 2000.67.0)

This has happened before with issue #43 (from 10 years ago) - I'm not sure if this is a similar linkage problem, or if it is related to the new Fortran-C interface i.e. these should be statically linked along with BLAS/LAPACK?

[Matt Denwood]

Update

With the following:

export LDFLAGS="/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgfortran.a /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgcc.a /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libquadmath.a -static-libgfortran"

The build succeeds, and the libgfortran.dylib dependency is gone (although I do get clang++: warning: argument unused during compilation: '-static-libgfortran' [-Wunused-command-line-argument]).

Unfortunately clang doesn't like -static-libgcc - I can manually remove -libquadmath from all Makefiles after running configure, but that seems like a hack - so it would be better to persuade configure that -libquadmath simply isn't needed.

A further problem is that src/terminal/.libs/jags-terminal ends up with a libltdl.dylib dependency - I am pretty sure this is also spurious, as replacing that with a dummy (blank) dylib does not break the build/check (although the binary still has the dependency on the dummy dylib).

So to fix this I think we need to:

1. Find a way of preventing configure adding -lquadmath (and -lgfortran) flags if the static libraries are also supplied.
2. Find out where the libltdl.dylib dependency is coming from.

I'll continue investigating, but comments/suggestions would be welcome!

[Matt Denwood]

Update 2

The configure option --with-included-ltdl solves the libltdl dependency.

There doesn't seem to be any neat way of preventing the -lquadmath flag being generated, but a manual find/sed step between configure and make resolves this (albeit in a hacky way). So the following works for static linkage to both gfortran (including libquadmath/libgcc) and openblas:

export PATH="/opt/gfortran/bin/:$PATH"
export LDFLAGS="-static-libgfortran"
export LIBS="/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgcc.a /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libquadmath.a"
./configure --with-included-ltdl --with-blas=*PATH*/OpenBLAS-0.3.30/libopenblas_vortex-r0.3.30.a
find . -name 'Makefile' -type f -exec sed -i '' -e 's/-lquadmath//' {} +
make -j 12 check

The resulting binaries have the following dynamic linkage, as expected:

  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 2000.67.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1356.0.0)

Unless anyone has a better way of addressing the -lquadmath issue on macOS (where -static-libgcc isn't an option) then let me know - otherwise I will add the instructions above to the installation manual (for generating standalone binaries).

[Martyn Plummer]

Sorry I should have spotted the --with-included-ltdl issue, but even I'm getting rusty on the configuration options. The command line interface uses libtool-ltdl to dynamically load the modules so the jags-terminal executable must be linked to either the internally provided library (e.g. on Windows) or the system one (e.g. on Linux). This is not necessary when using JAGS with R which has its own dynamic loading mechanism.

The libquadmath dependency is down to the fact that the JAGS library now includes some Fortran code. Most of this comes from the C-Fortran interface, but the JAGS library also now incorporates the CHOLMOD sparse matrix algebra library which also includes a small amount of Fortran code. When mixed C++/Fortran sources are compiled they must be linked with the C++ compiler, but the C++ compiler knows nothing about what flags are necessary for linking Fortran code. GNU autoconf provides a macro AC_FC_LIBRARY_LDFLAGS to provide suitable linker flags for Fortran code and this is what we use in JAGS (See configure.ac).

As far as I know, AC_FC_LIBRARY_LDFLAGS compiles a simple Fortran program with the Fortran compiler, then reads off the libraries it is linked to and infers the appropriate library flags to pass to the linker. These are added to FCLIBS which is then used at link time when the JAGS library is built, e.g. on my Fedora desktop:

FCLIBS=' -L/usr/lib/gcc/x86_64-redhat-linux/15 -L/usr/lib/gcc/x86_64-redhat-linux/15/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/15/../../.. -L/lib -L/usr/lib -lgfortran -lm -lquadmath'

This is not a problem on LInux because the user can always install libquadmath to satisfy this dependency. If the problem on Mac OS is creating a dependency to a library in the toolchain that will not be in the end-user's system then we need to find a way to statically link it instead.

[Matt Denwood]

We should certainly try to avoid the libgfortran and libquadmath runtime dependencies, as these are not part of a standard macOS install (they are installed as part of the GNU gfortran binary that Simon provides via https://mac.r-project.org/tools/). However, I think I am now doing that, albeit using a hack...

Using the following LDLAGS and LIBS:

export LDFLAGS="-static-libgfortran"
export LIBS="/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgcc.a /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libquadmath.a"

The configure script on my system now generates the following:

FCLIBS =  -L/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0 -L/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/../../.. -lemutls_w -lheapt_w /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgcc.a /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libquadmath.a /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgfortran.a -lquadmath

As we get libgcc.a/libquadmath.a/libgofrtran.a we have them statically linked - but apparently autoconf doesn't realise this is sufficient, and adds an unneeded -lquadmath, which results in dynamic linkage to libquadmath.dylib (in addition to the static linkage). I know it isn't needed because I can replace that dylib with an empty shared library and everything works fine - but the unnecessary runtime dependency remains. The only solution I have found is to use find/sed to obliterate all mention of -lquadmath from the Makefiles between running configure and make. This produces binaries that are free of the runtime dependencies, and pass make check ... although I still need to run more extensive tests to make sure there are no side-effects.

The only potential solution I have found is to use -static-libgcc (analagously to -static-libgfortran above, which succeeds in removing the -lgfortran flag) - but this is not supported by clang. So we seem to be stuck with my hack unless someone smarter than me can sweet-talk autoconf into recognising that -lquadmath should be omitted?

[Martyn Plummer]

Further investigation of the log file config.log shows that the original linker flags when the test Fortran program is compiled by the configure script were --as-needed -lquadmath. The --as-needed qualifier ensures that you don't link to libquadmath if you are not using any quad-precision floats. Unfortunately the qualifier doesn't make it into FCLIBS as you can see from my example above. So we get an unqualified link to libquadmath even though we are not using any quad-precision arithmetic.

One possible solution is to allow the user to provide their own Fortran library flags, e.g. by defining FCLIBS_USER at configure time. If this is defined then we don't call the autoconf macro that adds the unwanted library. Of course you would then need to get the linker flags exactly right by hand, but you could do this as follows:

1. Configure JAGS normally
2. Capture the FCFLAGS variable from the file config.log
3. Strip out the unwanted library and define this as FCLIBS_USER
4. reconfigure JAGS, then build

It's clunky but if you really don't want the libquadmath dependency then I can try it.

Another approach is to try to fix the AC_FC_LIBRARY_LDFLAGS macro so that it does pass the --as-needed flag when appropriate.

[Matt Denwood]

Sorry I didn't see this reply when I posted my message just now.

I think allowing FCLIBS_USER to override the autoconf macro is a good solution - at least much better than my using find/sed to remove the -lquadmath flag from all Makefiles manually. I already know what the correct linker flags are; I just can't get them to propogate through the configure script correctly. For standard compilation this is obviously a non-issue - it is just for creating portable binaries - so I think an additional environmental variable that only I need to worry about is a good compromise.

Having said that, if AC_FC_LIBRARY_LDFLAGS could be fixed that may be a cleaner solution?

[Martyn Plummer]

Last word on this before I sign off for now. If you supply FCLIBS at configure time then this will override the macro, e.g. on Linux I can do

FCLIBS="-lgfortran -lm" ./configure ...

and this build JAGS without linking to libquadmath (I have omitted a lot of -L/path/to/libs flags but it still seems to work).

However, you can still pick up a recursive dependency on libquadmath from a linked library. For example libflexiblas is linked to libquadmath on my Linux desktop so if I rely on flexiblas for my BLAS implementation then JAGS will also be linked to libquadmath..

[Matt Denwood]

Ah yes, if I set both:

FCLIBS="/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgcc.a /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libquadmath.a /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0/libgfortran.a"

And:

FFLAGS="-static-libgfortran -static-libquadmath"

Then this prevents -lquadmath being passed to the linker. The resulting binary has no runtime dependencies on gfortran/quadmath so that should be fine.

I'll add all this to the macOS specific section of the installation guide, along with instructions for the different BLAS/LAPACK options, when I am back in the office next week.