[Numpy-discussion] include file location and .eggs

[Andrew S. <str...@as...>]

(We've had threads about this before.  See
http://www.scipy.org/documentation/mailman?fn=scipy-dev/2005-September/003238.html
and
http://sourceforge.net/mailarchive/forum.php?thread_id=9474853&forum_id=4890)

There's been recent discussion regarding the location of the numpy
include files. The pre-numpy way (Numeric and numarray) was to use
distutils's install_headers, which places them in the long-familiar
locations like /usr/local/include/python2.3/ (on *nix). This was a
solution we all lived with for many years. Why then, is numpy rocking
the boat and putting the include files somewhere else?

For me, and I suspect for most others in favor of the new way, the
answer boils down to using eggs with out-of-the box numpy. Eggs are the
distributable form of Python package distributions made with setuptools
(see http://peak.telecommunity.com/DevCenter/setuptools ). This isn't an
email to convince everyone to use eggs, but I would like to say what
issues eggs solve, what they make possible, and why the current include
file location doesn't work with them. I _would_ like to convince people
that supporting eggs is a valid thing for numpy to do, even if it means
some change in the include file location and therefore some pain.
(Although the amount of pain seems minimal to me.)  I hope the
developers of numpy-based extension modules, even if they never plan on
using eggs, are willing to accomodate those of us who do.

= Why can't eggs use traditional include locations? =
Multiple version support. Using eggs, it is possible to have multiple
versions of the same distribution, such as numpy, installed within a
single Python installation. (I use "distribution" to mean collection of
Python packages and modules created by a call to distutils' setup() and
use that word because some distributions, such as matplotlib, include
multiple top-level packages.) This means I can have a root-installed,
site-wide installation of numpy on my unix machine for all users but I
can easily play with a per-user installed numpy installation if I want
to. With multiple versions of numpy installed on my system, the include
files I need to use to compile an extension module are obviously going
to depend on the right version of the headers. This simply isn't
possible using the traditional include location, but requires the
appropriate headers for the the version of numpy in use. I don't see any
easier solution to this issue than what eggs already has implemented --
import the numpy package and ask it where its include files are (which
will be in a sub-directory of it's top-level space within site-packages).

= What issues do eggs solve? =
I already mentioned multiple version support. eggs also offer dependency
analysis and resolution. They also allow compatibility with
easy_install, meaning automatic downloading and installation of
dependencies (see http://peak.telecommunity.com/DevCenter/EasyInstall ).

= What else do eggs make possible? =
Eggs are distributed as a single ".egg" file, which is really just a
specially formatted .zip file. If the package is "zip-safe", eggs are
installed in site-packages as a single zip file. Even if they're not
zip-safe, eggs are installed in site-packages in their own single
directory. This makes package management much easier.
Another of my favorite enhancements that setuptools brings to Python is
runtime plugin support. If I have package A, which can optionally use
plugins from module B, I can install A without installing B. When I
install B, A will automatically be able to use B. If I write C to the
same plugin interface that B uses, A can also use C, even if A never
knew of the existance of C. There is lots more that setuptools enables.
This is necessarily only a short list of what eggs can do and are useful
for. Please see http://peak.telecommunity.com/DevCenter/setuptools for
more information.

I hope those with reservations about the change in include file
locations now understand the reason for the change and, even more
optimistically, I hope those folks are willing to accept the current
state of affairs. I will attempt to answer any more questions on the issue.

Cheers!
Andrew