[Plplot-devel] python extension linking, was RE: python branch now "merged"

[Geoffrey F. <Geo...@at...>]

Geoffrey Furnish writes:
 > The other thing worth discussing is just to draw attention to the fact
 > that in the past I had trouble with how the python modules are being
 > linked.  I haven't gotten as far as reinvestigating that, so I'll post
 > more on it later.  But in the past I found that explicitly linking to
 > PYTHON_LIBRARIES caused problems for some applications.  So, that's an
 > open issue that I'll be reporting more on later, after I look into it
 > again.

Okay, I had some time this afternoon to press on to the next step, that of
trying to build an application that links against Python, PLplot, and various
other things, all bundled up into a "prefix".  Previously I was doing this
using the PLplot svn branches/python branch.  Now I'm trying to do it on
trunk, and without deleting the PYTHON_LIBRARIES element in the python
extension module linking rules.

Summary: It won't work.

Details:

Well, let's start with background.  My application is divided into
essentially two parts:  the "core", which is the part I am actively
developing, and the "prefix" which is the conglomeration of all the stuff
that the application links against, but which is not rightfully understood to
be part of the core application itself.  PLplot goes into the prefix, as do
the versions of Tcl, Tk, Python and half a dozen other packages (Xerces,
etc/more), that the application links against.  

Why jam all this stuff into an application-specific prefix?  Why not just
rely on picking up all that stuff from system directories (/usr/lib64, etc)?

Simple.  In one word: "control".  Meaning, both what I can personally
control, and what I cannot control.  I build my application against a very
carefully configured prefix of libraries because that way I know *exactly*
waht is in there, I qualify my application linked in that configuration, and
I distribute the whole shebang when I ship the application.  Both the core
and the prefix all go out in a coordinated way.  The application is qualified
in this operating condition, and it runs in the field with same run time
configuration of libraries, because the prefix goes along for the ride, and
the application is linked with -rpath so that it picks up packages from the
prefix, not from anywhere else.  That's the part I can control.

What I cannot control, is what is on the systems the application will run
on.  It might not have Tcl/Tk at all, or it might have different (either
older or newer) versions than the ones the application is qualified against.
Especially vexing is the situation where the destination computers do not
have all the needed packages.  And there is *nothing* I can do to influence
the system administrators to put *my* prefered package configuration on the
machines.  The computers are shared, my application is just one of
thousands.  So that's what I can't control.

So that's why this whole business of prefix configuration is so important to
me.  

Now, what I find is that if I configure Python (today I tested 2.6.2, but I
believe from prior experience that 2.5.x is the same in these regards) by
just saying:

     cd Python-2.6.2
     configure --prefix=$PREFIX
     make
     make install

Then what happens is that libpython2.6.a is installed in $PREFIX/lib, but
libpython2.6.so is NOT installed.

To get that installed, you have to do:

    cd Python-2.6.2
    make distclean     # Critical to avoid "configure hysteresis"
    configure --prefix=$PREFIX --enable-shared
    make
    make install

If you do that, then libpython2.6.so shows up in $PREFIX/lib.

HOWEVER: PLplot's Python binding needs numpy.  And here comes the rub.  It
turns out that Numpy's installation will *not work* if the python was built
--enable-shared.  If you don't build --enable-shared, then you can go into
the numpy distribution and run $PREFIX/bin/python setup.py install, and it
goes right where you want it, under $PREFIX.  But if your installed
$PREFIX/bin/python was configured with --enable-shared, then KABOOM, when you
run numpy's installer, it tries to write into /usr/lib64/*, and ignores the
PREFIX to which your python was configured.

Without numpy, there is no PLplot python binding.

Without leaving off --enable-shared, there is no installed numpy.

Without --enable-shared, there is no $PREFIX/lib/libpython2.6.so.

Without $PREFIX/lib/libpython2.6.so, there is no way to build the PLplot
python extension modules (if they're configured to link against
PYTHON_LIBRARIES). 

The only solution to this unholy mess that I have been able to find, is to
leave PYTHON_LIBRARIES off the link line for the PLplot python extension
modules.  On Linux at least, I know that this omission is inconsequential.
When you actually load the extension modules at run time, the symbols are
filled in from the dynloading executor process (which in my case will turn
out to be an embellished python with some application-specific python
extension stuff compiled in).  And all is well.

But I understand that other systems' dynloaders may work differently, and
that this may not work in general for other systems.

I hope the above rendition is persuasive in conveying that there is at least
one perfectly reasonable scenario in which a PLplot-based application
developer flatly requires the ability to build PLplot with its python
extension modules not explicitly linked to the libpython2.6.so.

What I do not know is if there are other circumstances, possibly involving
other operating platfomrs, which flatly require that explicit linkage to
libpython2.6.so IS used for the PLplot python extension modules.

Accordinly, I make the following modest proposal.

1) Tomorrow I will commit a change that takes PYTHON_LIBRARIES out of the
   explicit link lists for the python extension modules.

2) I'd like to ask people on non-Linux systems to test the python bindings
   and examples, to see whether they work without that explicit linkage.

3) Depending on the resulting reports, I could imagine some possible
   outcomes:

   A)  Maybe this will be okay.
   B)  Maybe we will conclude that it is always okay on certain systems, and
       always not okay on certain other systems. In this case we build more
       complex cmake logic to form the extension module linkage list in a way
       that varies by platform.

   C)  Another possible option would be to add a cmake variable which a user
       like me could set to some value to request the non-explicit linkage.
       I would guess that people would prefer the current status quo to
       remain the default.  And I will be satisifed if there is simply some
       way that I can document, which will get the behavior that I know is
       needed for at least some legitimate situations.

Comments or questions welcome.

-- 
Geoff