numpy-discussion

[Numpy-discussion] Setuptools leftover junk

[Fernando P. <fpe...@gm...>]

Hi all,

I recently noticed one of my in-house projects started leaving aroun
.egg-info directories after I ran its setup.py, even though I don't
use setuptools for anything at all.  For now I just added an extra
clean rule to my makefile and forgot about it, but it kind of annoyed
me.

Today I looked at the temp directory where I've been making my
numpy/scipy installs from SVN, and here's what I saw:

longs[site-packages]> d
/home/fperez/tmp/local/lib/python2.4/site-packages
total 228
drwxr-xr-x  2 fperez   4096 2006-06-21 22:16 dateutil/
drwxr-xr-x  7 fperez   4096 2006-06-28 02:50 matplotlib/
drwxr-xr-x 13 fperez   4096 2006-06-28 02:38 numpy/
drwxr-xr-x  2 fperez   4096 2006-06-21 21:28 numpy-0.9.9.2660-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-22 21:29 numpy-0.9.9.2665-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-24 11:33 numpy-0.9.9.2674-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-24 15:08 numpy-0.9.9.2675-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-25 12:40 numpy-0.9.9.2677-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-26 23:32 numpy-0.9.9.2691-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-28 02:38 numpy-0.9.9.2696-py2.4.egg-info/
-rw-r--r--  1 fperez     31 2006-03-18 20:11 pylab.py
-rw-r--r--  1 fperez    178 2006-06-24 13:29 pylab.pyc
drwxr-xr-x 20 fperez   4096 2006-06-28 11:20 scipy/
drwxr-xr-x  2 fperez   4096 2006-06-21 21:36 scipy-0.5.0.1990-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-22 21:36 scipy-0.5.0.1998-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-24 15:15 scipy-0.5.0.1999-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-25 12:46 scipy-0.5.0.2000-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-26 23:37 scipy-0.5.0.2004-py2.4.egg-info/
drwxr-xr-x  2 fperez   4096 2006-06-28 02:48 scipy-0.5.0.2012-py2.4.egg-info/


Is it really necessary to have all that setuptools junk left around,
for those of us who aren't asking for it explicitly?  My personal
opinions on setuptools aside, I think it's just a sane practice not to
create this kind of extra baggage unless explicitly requested.

I scoured my home directory for any .file which might be triggering
this inadvertedly, but I can't seem to find any, so I'm going to guess
this is somehow being caused by numpy's own setup.  If it's my own
mistake, I'll be happy to be shown how to coexist peacefully with
setuptools.

Since this also affects user code (I think via f2py or something
internal to numpy, since all I'm calling is f2py in my code), I really
think it would be nice to clean it.

Opinions?

f

Re: [Numpy-discussion] Setuptools leftover junk

[Robert K. <rob...@gm...>]

Fernando Perez wrote:

> Is it really necessary to have all that setuptools junk left around,
> for those of us who aren't asking for it explicitly?  My personal
> opinions on setuptools aside, I think it's just a sane practice not to
> create this kind of extra baggage unless explicitly requested.
> 
> I scoured my home directory for any .file which might be triggering
> this inadvertedly, but I can't seem to find any, so I'm going to guess
> this is somehow being caused by numpy's own setup.  If it's my own
> mistake, I'll be happy to be shown how to coexist peacefully with
> setuptools.
> 
> Since this also affects user code (I think via f2py or something
> internal to numpy, since all I'm calling is f2py in my code), I really
> think it would be nice to clean it.

numpy.distutils uses setuptools if it is importable in order to make sure that 
the two don't stomp on each other. It's probable that that test could probably 
be done with Andrew Straw's method:

   if 'setuptools' in sys.modules:
     have_setuptools = True
     from setuptools import setup as old_setup
   else:
     have_setuptools = False
     from distutils.core import setup as old_setup

Tested patches welcome.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Re: [Numpy-discussion] Setuptools leftover junk

[David M. C. <co...@ph...>]

On Wed, 28 Jun 2006 13:32:15 -0500
Robert Kern <rob...@gm...> wrote:

> Fernando Perez wrote:
> 
> > Is it really necessary to have all that setuptools junk left around,
> > for those of us who aren't asking for it explicitly?  My personal
> > opinions on setuptools aside, I think it's just a sane practice not to
> > create this kind of extra baggage unless explicitly requested.
> > 
> > I scoured my home directory for any .file which might be triggering
> > this inadvertedly, but I can't seem to find any, so I'm going to guess
> > this is somehow being caused by numpy's own setup.  If it's my own
> > mistake, I'll be happy to be shown how to coexist peacefully with
> > setuptools.
> > 
> > Since this also affects user code (I think via f2py or something
> > internal to numpy, since all I'm calling is f2py in my code), I really
> > think it would be nice to clean it.
> 
> numpy.distutils uses setuptools if it is importable in order to make sure
> that the two don't stomp on each other. It's probable that that test could
> probably be done with Andrew Straw's method:
> 
>    if 'setuptools' in sys.modules:
>      have_setuptools = True
>      from setuptools import setup as old_setup
>    else:
>      have_setuptools = False
>      from distutils.core import setup as old_setup
> 
> Tested patches welcome.

Done. I've also added a 'setupegg.py' module that wraps running 'setup.py'
with an import of setuptools (it's based on the one used in matplotlib).

easy_install still works, also.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

Re: [Numpy-discussion] Setuptools leftover junk

[Fernando P. <fpe...@gm...>]

On 6/28/06, David M. Cooke <co...@ph...> wrote:

> Done. I've also added a 'setupegg.py' module that wraps running 'setup.py'
> with an import of setuptools (it's based on the one used in matplotlib).
>
> easy_install still works, also.

You beat me to it :)

However, your patch has slightly different semantics from mine: if
bdist_egg fails to import, the rest of setuptools is still used.  I
don't know if that's safe.  My patch would consider /any/ failure in
the setuptools imports as a complete setuptools failure, and revert
out to basic distutils.

Let me know if you want me to put in my code instead, here's a patch
from my code against current svn (after your patch), in case you'd
like to try it out.

Cheers,

f



Index: core.py
===================================================================
--- core.py     (revision 2701)
+++ core.py     (working copy)
@@ -1,20 +1,30 @@
-
 import sys
 from distutils.core import *

-if 'setuptools' in sys.modules:
-    have_setuptools = True
-    from setuptools import setup as old_setup
-    # easy_install imports math, it may be picked up from cwd
-    from setuptools.command import develop, easy_install
+# Don't pull setuptools in unless the user explicitly requests by having it
+# imported (Andrew's trick).
+have_setuptools = 'setuptools' in sys.modules
+
+# Even if setuptools is in, do a few things carefully to make sure the version
+# is recent enough to have everything we need before assuming we can proceed
+# using setuptools throughout
+if have_setuptools:
     try:
-        # very old versions of setuptools don't have this
+        from setuptools import setup as old_setup
+        # very old setuptools don't have this
         from setuptools.command import bdist_egg
+        # easy_install imports math, it may be picked up from cwd
+        from setuptools.command import develop, easy_install
     except ImportError:
+        # Any failure here is probably due to an old or broken setuptools
+        # leftover in sys.modules, so treat it as if it simply weren't
+        # available.
         have_setuptools = False
-else:
+
+# If setuptools was flagged as unavailable due to import problems, we need the
+# basic distutils support
+if not have_setuptools:
     from distutils.core import setup as old_setup
-    have_setuptools = False

 from numpy.distutils.extension import Extension
 from numpy.distutils.command import config

Re: [Numpy-discussion] Setuptools leftover junk

[David M. C. <co...@ph...>]

On Wed, 28 Jun 2006 13:18:35 -0600
"Fernando Perez" <fpe...@gm...> wrote:

> On 6/28/06, David M. Cooke <co...@ph...> wrote:
> 
> > Done. I've also added a 'setupegg.py' module that wraps running 'setup.py'
> > with an import of setuptools (it's based on the one used in matplotlib).
> >
> > easy_install still works, also.
> 
> You beat me to it :)
> 
> However, your patch has slightly different semantics from mine: if
> bdist_egg fails to import, the rest of setuptools is still used.  I
> don't know if that's safe.  My patch would consider /any/ failure in
> the setuptools imports as a complete setuptools failure, and revert
> out to basic distutils.

Note that your patch will still import setuptools if the import of bdist_egg
fails. And you can't get around that by putting the bdist_egg import first,
as that imports setuptools first.

(I think bdist_egg was added sometime after 0.5; if your version of
setuptools is *that* old, you'd be better off not having it installed.)

The use of setuptools by numpy.distutils is in two forms: explicitly
(controlled by this patch of code), and implicitly (because setuptools goes
and patches distutils). Disabling the explicit use won't actually fix your
problem with the 'install' command leaving .egg_info directories (which,
incidentally, are pretty small), as that's done by the implicit behaviour.

[Really, distutils sucks. I think (besides refactoring) it needs it's API
documented better, or least good conventions on where to hook into.
setuptools and numpy.distutils do their best, but there's only so much you
can do before everything goes fragile and breaks in unexpected ways.]

With the "if 'setuptools' in sys.modules" test, if you *are* using
setuptools, you must have explicitly requested that, and so I think a failure
on import of setuptools shouldn't be silently passed over.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

Re: [Numpy-discussion] Setuptools leftover junk

[Fernando P. <fpe...@gm...>]

On 6/28/06, David M. Cooke <co...@ph...> wrote:
> On Wed, 28 Jun 2006 13:18:35 -0600
> "Fernando Perez" <fpe...@gm...> wrote:
>
> > On 6/28/06, David M. Cooke <co...@ph...> wrote:
> >
> > > Done. I've also added a 'setupegg.py' module that wraps running 'setup.py'
> > > with an import of setuptools (it's based on the one used in matplotlib).
> > >
> > > easy_install still works, also.
> >
> > You beat me to it :)
> >
> > However, your patch has slightly different semantics from mine: if
> > bdist_egg fails to import, the rest of setuptools is still used.  I
> > don't know if that's safe.  My patch would consider /any/ failure in
> > the setuptools imports as a complete setuptools failure, and revert
> > out to basic distutils.
>
> Note that your patch will still import setuptools if the import of bdist_egg
> fails. And you can't get around that by putting the bdist_egg import first,
> as that imports setuptools first.

Well, but that's still done after the 'if "setuptools" in sys.modules'
check, just like yours.  The only difference is that my patch treats a
later failure as a complete failure, and reverts out to old_setup
being pulled out of plain distutils.

> (I think bdist_egg was added sometime after 0.5; if your version of
> setuptools is *that* old, you'd be better off not having it installed.)

Then it's probably fine to leave it either way, as /in practice/ the
two approaches will produce the same results.

> The use of setuptools by numpy.distutils is in two forms: explicitly
> (controlled by this patch of code), and implicitly (because setuptools goes
> and patches distutils). Disabling the explicit use won't actually fix your
> problem with the 'install' command leaving .egg_info directories (which,
> incidentally, are pretty small), as that's done by the implicit behaviour.

It's not their size that matters, it's just that I don't like tools
littering around with stuff I didn't ask for.  Yes, I like my code
directories tidy ;)

> [Really, distutils sucks. I think (besides refactoring) it needs it's API
> documented better, or least good conventions on where to hook into.
> setuptools and numpy.distutils do their best, but there's only so much you
> can do before everything goes fragile and breaks in unexpected ways.]

I do hate distutils, having fought it for a long time.  Its piss-poor
dependency checking is one of its /many/ annoyances.  For a package
with as long a compile time as scipy, it really sucks not to be able
to just modify random source files and trust that it will really
recompile what's needed (no more, no less).

Anyway, thanks for heeding this one.  Hopefully one day somebody will
do the (painful) work of replacing distutils with something that
actually works (perhaps using scons for the build engine...)  Until
then, we'll trod along with massively unnecessary rebuilds :)

Cheers,

f

Re: [Numpy-discussion] Setuptools leftover junk

[David M. C. <co...@ph...>]

On Wed, 28 Jun 2006 13:46:07 -0600
"Fernando Perez" <fpe...@gm...> wrote:

> On 6/28/06, David M. Cooke <co...@ph...> wrote:
> 
> > [Really, distutils sucks. I think (besides refactoring) it needs it's API
> > documented better, or least good conventions on where to hook into.
> > setuptools and numpy.distutils do their best, but there's only so much you
> > can do before everything goes fragile and breaks in unexpected ways.]
> 
> I do hate distutils, having fought it for a long time.  Its piss-poor
> dependency checking is one of its /many/ annoyances.  For a package
> with as long a compile time as scipy, it really sucks not to be able
> to just modify random source files and trust that it will really
> recompile what's needed (no more, no less).
> 
> Anyway, thanks for heeding this one.  Hopefully one day somebody will
> do the (painful) work of replacing distutils with something that
> actually works (perhaps using scons for the build engine...)  Until
> then, we'll trod along with massively unnecessary rebuilds :)

I've tried using SCons -- still don't like it. It's python, but it's too
unpythonic for me. (The build engine itself is probably fine, though.)

A complete replacement for distutils isn't needed: bits and pieces can be
replaced at a time (it gets harder if you've got two packages like setuptools
and numpy.distutils trying to improve it, though). For instance, the
CCompiler class could be replaced in whole with a rewrite, keeping what could
be considered the public API. I've done this before with a version of
UnixCCompiler that let me specify a "toolchain": which C compiler and C++
compiler worked together, which linker to use for them, and associated flags.

I'm working (slowly) on a rewrite of commands/build_ext.py in numpy.distutils
that should keep track of source dependencies better, for instance.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

Re: [Numpy-discussion] Setuptools leftover junk

[Bryce H. <bhe...@en...>]

David M. Cooke wrote:
>
>>> [Really, distutils sucks. I think (besides refactoring) it needs it's API
>>> documented better, or least good conventions on where to hook into.
>>> setuptools and numpy.distutils do their best, but there's only so much you
>>> can do before everything goes fragile and breaks in unexpected ways.]
>>>       
>> I do hate distutils, having fought it for a long time.  Its piss-poor
>> dependency checking is one of its /many/ annoyances.  For a package
>> with as long a compile time as scipy, it really sucks not to be able
>> to just modify random source files and trust that it will really
>> recompile what's needed (no more, no less).
>>
>> Anyway, thanks for heeding this one.  Hopefully one day somebody will
>> do the (painful) work of replacing distutils with something that
>> actually works (perhaps using scons for the build engine...)  Until
>> then, we'll trod along with massively unnecessary rebuilds :)
>>     
>
> I've tried using SCons -- still don't like it. It's python, but it's too
> unpythonic for me. (The build engine itself is probably fine, though.)
>   
Agreed, last time I used it was almost a year ago, so it might have 
changed, but SCons does a quasi-2 stage build that is very unnatural. If 
you have python code nested between 2 build events, the python code is 
executed and the build events are queued.

BUT- its dependency management is great. Distutils suffers from 2 major 
problems as far as I am concerned: setup.py files often contain way too 
much business logic and verb-age for casual python developers, and 
worst-in-class dependency checking.

I've been considering moving all Enthought projects to SCons. If another 
large project, such as scipy were to go that way it would make my 
decision much simpler.

Bryce

Re: [Numpy-discussion] Setuptools leftover junk

[Fernando P. <fpe...@gm...>]

On 6/28/06, Robert Kern <rob...@gm...> wrote:

> numpy.distutils uses setuptools if it is importable in order to make sure that
> the two don't stomp on each other. It's probable that that test could probably
> be done with Andrew Straw's method:
>
>    if 'setuptools' in sys.modules:
>      have_setuptools = True
>      from setuptools import setup as old_setup
>    else:
>      have_setuptools = False
>      from distutils.core import setup as old_setup
>
> Tested patches welcome.

Well, tested as in 'I wrote a unittest for installation', no.  But
tested as in 'I built numpy, scipy, matplotlib, and my f2py-using
code', yes.  They all build/install fine, and no more *egg-info
directories are strewn around.

If this satisfies your 'tested patches', the code is:

Index: numpy/distutils/core.py
===================================================================
--- numpy/distutils/core.py     (revision 2698)
+++ numpy/distutils/core.py     (working copy)
@@ -1,16 +1,30 @@
-
 import sys
 from distutils.core import *
-try:
-    from setuptools import setup as old_setup
-    # very old setuptools don't have this
-    from setuptools.command import bdist_egg
-    # easy_install imports math, it may be picked up from cwd
-    from setuptools.command import develop, easy_install
-    have_setuptools = 1
-except ImportError:
+
+# Don't pull setuptools in unless the user explicitly requests by having it
+# imported (Andrew's trick).
+have_setuptools = 'setuptools' in sys.modules
+
+# Even if setuptools is in, do a few things carefully to make sure the version
+# is recent enough to have everything we need before assuming we can proceed
+# using setuptools throughout
+if have_setuptools:
+    try:
+        from setuptools import setup as old_setup
+        # very old setuptools don't have this
+        from setuptools.command import bdist_egg
+        # easy_install imports math, it may be picked up from cwd
+        from setuptools.command import develop, easy_install
+    except ImportError:
+        # Any failure here is probably due to an old or broken setuptools
+        # leftover in sys.modules, so treat it as if it simply weren't
+        # available.
+        have_setuptools = False
+
+# If setuptools was flagged as unavailable due to import problems, we need the
+# basic distutils support
+if not have_setuptools:
     from distutils.core import setup as old_setup
-    have_setuptools = 0

 from numpy.distutils.extension import Extension
 from numpy.distutils.command import config



May I?

keeping-the-world-setuptools-free-one-script-at-a-time-ly yours,

f