numpy-discussion

Re: [Numpy-discussion] RPMs out of date, have problems

[<gve...@la...>]

> 
> > I continue to be less than impressed by RPMs. Every time I try one I get
> > version conflicts with other software.
>
Agreed, I always use RPMs because they help me to keep my systems
in a reversible state. BUT, I never install binary RPMs that are
not taylored to the Distro-Version that I am running (Try to install
RH-7.2 packages on a RH-7.0). I may take source RPMs from other
distributions as a starting point for my own tweaks, but I NEVER
install 'foreign' RPMs.


Attached you'll find a replacement for the setup.py script allowing
to build a single binary and source RPM for Numeric + subpackages.

The README.RPM for a normal user would be something like:
(1) unpack python-numeric-XXX.YY.Z.tar.gz
(2) cd python-numeric-XXX.YY.Z
(3) python setup.py bdist_rpm
This will build a binary and source RPM adapted to YOUR system
(even yours, Paul, with your non-standard Python location) in
the subdirectory dist (to be created automatically)
(4) install with rpm -Uvh python-numeric-XXX.YY.Z-1.i?86.rpm
You can always uninstall with rpm -e python-numeric.

IMHO, this is a better way than trying to distibute binary RPMs.


It happens that my favorite distribution is not on JH's list :-(

But also, http://www.mandrakesoft.com/products/81/gaming-edition ,
the favorite toy of 11 years old whizkids, doing pygame-NumPy
programming and who are certainly future students of astronomy.


If you want to test the script:
(1) replace the old setup.py with mine
(2) rename Lib/Numeric.py to Lib/__init__.py (Numeric will become
a real Python package).
(3) python setup.py bdist
(4) and install the RPM

Caveats:
(1) renaming Numeric to python-numeric has as consequence that
the header files go into .../include/python2.1/python-numeric
instead of .../include/python2.1/Numeric. Personally I don't
like it (being author of a package that builds on Numeric).
(I think this can be fixed, if needed).

(2) I tried to acknowledge Paul Dubois' contributions in a
long_description (the subpackages do not build to separate
packages anymore).

(3) The setup.py script shows how to add documentation
to an RPM (requires to work around a bug in distutils).
Personnally, I think that HTML and PDF documentation
should also be added. In the current state of distutils
this requires inclusion of the documentation in the
python-numeric-XXX.YY.Z.tar.gz.


Gerard


---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/

Re: [Numpy-discussion] RPMs out of date, have problems

[<gve...@la...>]

> gve...@la... writes:
> 
> > Caveats:
> > (1) renaming Numeric to python-numeric has as consequence that
> > the header files go into .../include/python2.1/python-numeric
> > instead of .../include/python2.1/Numeric. Personally I don't
> > like it (being author of a package that builds on Numeric).
> 
> Me neither. But the fix is simple: take the distutils-generated spec
> file, edit the package name, and put it into the tar ball. The added
> advantage is that the binary RPM can be generated by rpm --rebuild
> numpy_xxx.tar.gz.
> 
You mean rpm -ta numpy_xxx.tar.gz

Disadvantage: by default, the package will go into a distro dependent place
(Mandrake, Red Hat and Suse are all different) and the package builder
has to have write access to that place (Suse-7.3 grants it, Mandrake not,
Red Hat don't know). This invites to building packages as root (DON'T).
Building packages as a normal user protects you from erratic
writes in /usr, /, ...

Remember, we want to satisfy the needs of astronomer newbies. Users like
you will allways find a solution.

python setup.py bdist_rpm needs only write access to the user's home
directory.

Of course, adding a generic spec file is the easy 2 minutes solution,
but I have choosen not to do that because it will invite bad policy
(providing binary RPMs is asking for trouble).

IMHO, the current setup.py/setup_all.py are not permitting distutils's
way of 'RPM building'. It is better to fix that.

I will try to see renaming the package from Numeric to python-numeric
allows to install the headers in ../python2.1/Numeric instead of
./python2.1/python-numeric by changing my setup.py file.

Gerard



---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/

Re: [Numpy-discussion] RPMs out of date, have problems

[Konrad H. <hi...@cn...>]

> Disadvantage: by default, the package will go into a distro dependent place
> (Mandrake, Red Hat and Suse are all different) and the package builder
> has to have write access to that place (Suse-7.3 grants it, Mandrake not,
> Red Hat don't know). This invites to building packages as root (DON'T).

RedHat doesn't by default (I changed that on my system).

> python setup.py bdist_rpm needs only write access to the user's home
> directory.

Everywhere? It didn't when I first tried (that was an early distutils
release under RedHat 6.2).

> Of course, adding a generic spec file is the easy 2 minutes solution,
> but I have choosen not to do that because it will invite bad policy
> (providing binary RPMs is asking for trouble).

And yet many people will not accept anything but binary RPMs, out of
ignorance or fear. We won't make everyone happy...

I admit that I never looked into fully automatic RPM generation by
distutils as I usually need to add some manual steps to the
build/install process. For example, distutils doesn't let me specify
compiler options, but I want time-critical code to be compiled with
the highest optimization level. Hopefully this will be addressed in
distutils one day.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] RPMs out of date, have problems

[<gve...@la...>]

> 
> > python setup.py bdist_rpm needs only write access to the user's home
> > directory.
> 
> Everywhere? It didn't when I first tried (that was an early distutils
> release under RedHat 6.2).
>
Can you check which version of Python/distutils. If it is a version python
that does not come with distutils, we can count it out.
> 
> > Of course, adding a generic spec file is the easy 2 minutes solution,
> > but I have choosen not to do that because it will invite bad policy
> > (providing binary RPMs is asking for trouble).
> 
> And yet many people will not accept anything but binary RPMs, out of
> ignorance or fear. We won't make everyone happy...
>
Yes, and they will complain to JH if they are failing to install JH's
binary RPMs. The best we can do is explain the motivation of NOT
providing binary RPMs. If that is not sufficient for some, we can
only advise that they switch to Windows or pester their distro
providers for RPMs of the latest NumPy.
> 
> I admit that I never looked into fully automatic RPM generation by
> distutils as I usually need to add some manual steps to the
> build/install process. For example, distutils doesn't let me specify
> compiler options, but I want time-critical code to be compiled with
> the highest optimization level. Hopefully this will be addressed in
> distutils one day.
>
There are two ways to do this:

(1) Extension( ..., extra_compile_args = [ "-O17" ], ...)
This will append the optimization level 17 to the compiler flag
(the last -O option counts).

(2) export CFLAGS=-O3; python setup.py build
This will append -O3

The result of (1) and (2) is gcc .... -017 -03


Gerard

PS: distutils may look intimidating at first, but it is really
simple compared to automake/autoconf. The disadvantage is that
it is not very well tested.


---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/

Re: [Numpy-discussion] RPMs out of date, have problems

[Konrad H. <hi...@cn...>]

> Can you check which version of Python/distutils. If it is a version python
> that does not come with distutils, we can count it out.

I don't remember, and I don't have any of that stuff left.

> Yes, and they will complain to JH if they are failing to install JH's
> binary RPMs. The best we can do is explain the motivation of NOT
> providing binary RPMs. If that is not sufficient for some, we can

Somebody else will provide them, and you will still get complaints. I
also get them. Fortunately there is a standard answer to send out.

> There are two ways to do this:
> 
> (1) Extension( ..., extra_compile_args = [ "-O17" ], ...)
> This will append the optimization level 17 to the compiler flag
> (the last -O option counts).

But that will add the argument for all platforms. I need to specify
compilation flags separately for each platform/compiler. I might even
want to compile some modules with the new Intel compiler for Linux.

> (2) export CFLAGS=-O3; python setup.py build
> This will append -O3

For all extension modules - I need to specify different flags for each
extension module, in one case even for a single source code file.

My current solution is to build everything with distutils and then
recompile the critical modules manually in the spec file.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] RPMs out of date, have problems

[eric <er...@en...>]

> > (1) Extension( ..., extra_compile_args = [ "-O17" ], ...)
> > This will append the optimization level 17 to the compiler flag
> > (the last -O option counts).
>
> But that will add the argument for all platforms. I need to specify
> compilation flags separately for each platform/compiler. I might even
> want to compile some modules with the new Intel compiler for Linux.
>
> > (2) export CFLAGS=-O3; python setup.py build
> > This will append -O3
>
> For all extension modules - I need to specify different flags for each
> extension module, in one case even for a single source code file.

This seems to be one of the big complaints about distutils for scientist
(and maybe others).  I think it is also one of Paul Dubois' major reasons
for not liking distutils for Fortran files.  I've run into this occasionally
too and wished for a dictionary argument, maybe file_special_instructions,
that had file names as keys and a dictionary of options for that file as the
value.  I think this might go some distance to solving the problem.

file_special_instructions = {
                                           'really_fast.c': { compiler =
'icc',

override_compiler_flags = ['O2', 'tpp7']
                                                                   }
                                           'sorta_fast.c':  {
extra_compile_args = ['-O2']}
                                           }

So an extension module that used this and had 'slow.c', 'really_fast.c', and
'kinda_fast.c' would use the standard compile tools for 'slow.c', use the
specified compiler with the specified flags for 'really_fast.c', and add the
specified compiler flags to the standard flags for 'sorta_fast.c'.  This
approach would provide quite a bit of flexiblity.  To handle platform
specific versions of file_special_instructions, I think you'd have to use
if/thens within the setup.py file to build a separate set of instructions
for each platform.

There is one other major issue I've run into with distutils that I haven't
found a way around.  Sometimes you need compiler/linker flags inter-mingled
with source or library files in a certain way.  This can occur on SunOS when
you want to link statically to some libraries and dynamically to others.
distutils just doesn't provide a way of doing this that I have found.

SciPy now has a package of extensions/changes to distutils called
scipy_distutils helpful for building fortran based extension modules and
other things needed by SciPy.  We could experiment with adding something
like the file_special_instructions flag if others thought it useful.  Later,
it could be folded back into distutils if the rest of the community wanted
it.

Thoughts?

eric

Re: [Numpy-discussion] RPMs out of date, have problems

[Konrad H. <hi...@cn...>]

gve...@la... writes:

> Caveats:
> (1) renaming Numeric to python-numeric has as consequence that
> the header files go into .../include/python2.1/python-numeric
> instead of .../include/python2.1/Numeric. Personally I don't
> like it (being author of a package that builds on Numeric).

Me neither. But the fix is simple: take the distutils-generated spec
file, edit the package name, and put it into the tar ball. The added
advantage is that the binary RPM can be generated by rpm --rebuild
numpy_xxx.tar.gz.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------