numpy-discussion

[Numpy-discussion] .debs of numpy-0.9.8 available for Ubuntu Dapper

[Andrew S. <str...@as...>]

I've put together some .debs for numpy-0.9.8. There are binaries 
compiled for amd64 and i386 architectures of Ubuntu Dapper, and I 
suspect these will build from source for just about any Debian-based 
distro and architecture.

The URL is http://sefton.astraw.com/ubuntu/dapper and you would add the 
following lines to your /etc/apt/sources.list:
deb http://sefton.astraw.com/ubuntu/ dapper/
deb-src http://sefton.astraw.com/ubuntu/ dapper/

Although this is the culmination of my first serious attempt Debianizing 
something, I've attempted to build these "properly" (using inspiration 
from Matthias Klose's Numeric and numarray packages for Debian and 
Ubuntu, although I've updated the build system to use CDBS).

The numpy source has a build dependency on setuptools (0.6b2), which is 
also available at the repository. Numpy doesn't get installed as an 
.egg, but it carries along .egg-info, which means that numpy can be part 
of a setuptools dependency specification. This was done using the 
--single-version-externally-managed command for setuptools.

I'm building this repository to serve some of my needs at work, and I 
hope to add recent versions of several other projects including 
matplotlib and scipy in the coming days. I hope to be able to keep the 
repository up-to-date over time and to respond to bug reports and 
questions, although the amount of time I have to devote to this sort of 
stuff is unfortunately often near zero.

If I get some positive feedback, I'm likely to add this to the scipy.org 
download page. Also, I hope the official Debian and Ubuntu distros pick 
up numpy soon, and perhaps this will speed them along.

Re: [Numpy-discussion] .debs of numpy-0.9.8 available for Ubuntu Dapper

[Arnd B. <arn...@we...>]

Hi Andrew,

first thanks a lot for your effort - I am certain it
will be very much appreciated!

On Thu, 8 Jun 2006, Andrew Straw wrote:

> I've put together some .debs for numpy-0.9.8. There are binaries
> compiled for amd64 and i386 architectures of Ubuntu Dapper, and I
> suspect these will build from source for just about any Debian-based
> distro and architecture.
>
> The URL is http://sefton.astraw.com/ubuntu/dapper and you would add the
> following lines to your /etc/apt/sources.list:
> deb http://sefton.astraw.com/ubuntu/ dapper/
> deb-src http://sefton.astraw.com/ubuntu/ dapper/
>
> Although this is the culmination of my first serious attempt Debianizing
> something, I've attempted to build these "properly" (using inspiration
> from Matthias Klose's Numeric and numarray packages for Debian and
> Ubuntu, although I've updated the build system to use CDBS).
>
> The numpy source has a build dependency on setuptools (0.6b2), which is
> also available at the repository. Numpy doesn't get installed as an
> .egg, but it carries along .egg-info, which means that numpy can be part
> of a setuptools dependency specification. This was done using the
> --single-version-externally-managed command for setuptools.
>
> I'm building this repository to serve some of my needs at work, and I
> hope to add recent versions of several other projects including
> matplotlib and scipy in the coming days. I hope to be able to keep the
> repository up-to-date over time and to respond to bug reports and
> questions, although the amount of time I have to devote to this sort of
> stuff is unfortunately often near zero.

Alright, let's start with the first question:
We are still running debian sarge and therefore would have to
build the above from source.
I used the following steps:
- put
    deb-src http://sefton.astraw.com/ubuntu/ dapper/
  into /etc/apt/sources.list
- apt-get update  # update the source package search list
- apt-get source python-numpy
- cd python-numpy-0.9.8/
  dpkg-buildpackage -rfakeroot

and get:

dpkg-buildpackage: source package is python-numpy
dpkg-buildpackage: source version is 0.9.8-0ads1
dpkg-buildpackage: source maintainer is Andrew Straw <str...@as...>
dpkg-buildpackage: host architecture is i386
dpkg-checkbuilddeps: Unmet build dependencies: cdbs (>= 0.4.23-1.1)
build-essential python2.4-dev python-setuptools (>= 0.6b2)
python2.3-setuptools (>= 0.6b2) python2.4-setuptools (>= 0.6b2)
dpkg-checkbuilddeps: Build conflicts: atlas3-base
dpkg-buildpackage: Build dependencies/conflicts unsatisfied; aborting.
dpkg-buildpackage: (Use -d flag to override.)

What worries me is
a) the Build conflicts: atlas3-base
b) and the python2.3-dev *and* python2.4-dev dependency
Clearly, python-setuptools and cdbs are not yet installed
on my system (should be no problem).

> If I get some positive feedback, I'm likely to add this to the scipy.org
> download page. Also, I hope the official Debian and Ubuntu distros pick
> up numpy soon, and perhaps this will speed them along.

yes - that would be brilliant!

What about scipy: presently debian sarge comes with
scipy 0.3.2. Installing old-scipy and new-scipy side-by side
seems impossible (unless one does something like wxversion select
stuff...) - should the new scipy debs just replace the old ones?

Best, Arnd

Re: [Numpy-discussion] .debs of numpy-0.9.8 available for Ubuntu Dapper

[Pau G. <pau...@gm...>]

On 6/8/06, Andrew Straw <str...@as...> wrote:
> I've put together some .debs for numpy-0.9.8. There are binaries
> compiled for amd64 and i386 architectures of Ubuntu Dapper, and I
> suspect these will build from source for just about any Debian-based
> distro and architecture.
>
> The URL is http://sefton.astraw.com/ubuntu/dapper and you would add the
> following lines to your /etc/apt/sources.list:
> deb http://sefton.astraw.com/ubuntu/ dapper/
> deb-src http://sefton.astraw.com/ubuntu/ dapper/
>
> Although this is the culmination of my first serious attempt Debianizing
> something, I've attempted to build these "properly" (using inspiration
> from Matthias Klose's Numeric and numarray packages for Debian and
> Ubuntu, although I've updated the build system to use CDBS).
>
> The numpy source has a build dependency on setuptools (0.6b2), which is
> also available at the repository. Numpy doesn't get installed as an
> .egg, but it carries along .egg-info, which means that numpy can be part
> of a setuptools dependency specification. This was done using the
> --single-version-externally-managed command for setuptools.
>
> I'm building this repository to serve some of my needs at work, and I
> hope to add recent versions of several other projects including
> matplotlib and scipy in the coming days. I hope to be able to keep the
> repository up-to-date over time and to respond to bug reports and
> questions, although the amount of time I have to devote to this sort of
> stuff is unfortunately often near zero.
>
> If I get some positive feedback, I'm likely to add this to the scipy.org
> download page. Also, I hope the official Debian and Ubuntu distros pick
> up numpy soon, and perhaps this will speed them along.
>

cool, debian packages will be great, thanks!!

is your effort somehow related to
http://packages.debian.org/experimental/python/python2.3-numpy
?

it is a bit out of date, but already in experimental.

cheers,
pau

Re: [Numpy-discussion] .debs of numpy-0.9.8 available for Ubuntu Dapper

[Andrew S. <str...@as...>]

Pau Gargallo wrote:
> is your effort somehow related to
> http://packages.debian.org/experimental/python/python2.3-numpy
> ?
>
> it is a bit out of date, but already in experimental.
>   
I did have a look at their packaging infrastructure. It was breaking for 
me with numpy-0.9.8, so I started my debian/rules from scratch (and 
tried several methods along the way -- both debhelper and cdbs based). 
Now, upon re-looking at their debian/rules which is also cdbs based, I 
can see they have some nice code I should use (regarding installation of 
documentation and f2py). I'll try to integrate their changes into my 
next release. At that point I may simply be maintaining a more 
up-to-date version of theirs.

They also package new scipy. I'll see if I can leverage their efforts 
when I try to package that.

Re: [Numpy-discussion] .debs of numpy-0.9.8 available for Ubuntu Dapper

[Andrew S. <str...@as...>]

Arnd Baecker wrote:
> What worries me is
> a) the Build conflicts: atlas3-base
>   
I hoped to investigate further and post afterwards, but my preliminary 
findings that led to this decision are:

1) building with atlas (atlas3-base and atlas3-base-dev) caused a 
significant slowdown (~10x) on my simple test on amd64 arch:

import timeit
shape = '(40,40)'
timeit.Timer('a=ones(shape=%s);svd(a)'%shape,'from numpy import ones; 
from numpy.linalg import svd')
print "NumPy: ", t2.repeat(5,500)

2) Even having atlas installed (atlas3-base on amd64) caused a 
significant slowdown (~2x) on that test. This was similar to the case 
for i386, where I installed atlas3-sse2.
3) This is done in the source packages by Matthias Klose for both 
numeric and numarray, too. I figured he knows what he's doing.
> b) and the python2.3-dev *and* python2.4-dev dependency
>   
This is a _build_ dependency. The source package builds python 
python2.3-numpy and python2.4-numpy, so it needs Python.h for both.
> Clearly, python-setuptools and cdbs are not yet installed
> on my system (should be no problem).
>   
I hope the setuptools issue, in particular, does not present a problem. 
As I said, I have created this repository for work, and I find 
setuptools to be invaluable for maintaining order amongst all the Python 
packages I use internally. In any case, this is again only a build 
dependency -- all it does is creates a numpy-0.9.8-py2.x.egg-info 
directory in site-packages alongside numpy.

Let me be clear, since there's a lot of trepidation regarding 
setuptools: there is no use of setuptools (or even installation of 
setuptools) required to use these packages. Setuptools is required only 
to build from source.
>> If I get some positive feedback, I'm likely to add this to the scipy.org
>> download page. Also, I hope the official Debian and Ubuntu distros pick
>> up numpy soon, and perhaps this will speed them along.
>>     
>
> yes - that would be brilliant!
>   
OK, I'll wait a couple of days for some positive confirmation that this 
stuff works, (even from the various systems I'm setting up this 
repository for), and then I'll post it on the website.
> What about scipy: presently debian sarge comes with
> scipy 0.3.2. Installing old-scipy and new-scipy side-by side
> seems impossible (unless one does something like wxversion select
> stuff...) - should the new scipy debs just replace the old ones?
>   
Unless you do some apt-pinning, I think any new scipy (0.4.x) in any 
repository in your sources list will automatically override the old 
(0.3.x) simply via the versioning mechanisms of apt-get. I like the idea 
of a wxversion-alike, but I've shifted all my code to use numpy and the 
new scipy, so I don't have any motivation to do any implementation.

Re: [Numpy-discussion] .debs of numpy-0.9.8 available for Ubuntu Dapper

[Arnd B. <arn...@we...>]

On Thu, 8 Jun 2006, Andrew Straw wrote:

> Arnd Baecker wrote:
> > What worries me is
> > a) the Build conflicts: atlas3-base
> >
> I hoped to investigate further and post afterwards, but my preliminary
> findings that led to this decision are:
>
> 1) building with atlas (atlas3-base and atlas3-base-dev) caused a
> significant slowdown (~10x) on my simple test on amd64 arch:
>
> import timeit
> shape = '(40,40)'
> timeit.Timer('a=ones(shape=%s);svd(a)'%shape,'from numpy import ones;
> from numpy.linalg import svd')
> print "NumPy: ", t2.repeat(5,500)
>
> 2) Even having atlas installed (atlas3-base on amd64) caused a
> significant slowdown (~2x) on that test. This was similar to the case
> for i386, where I installed atlas3-sse2.
> 3) This is done in the source packages by Matthias Klose for both
> numeric and numarray, too. I figured he knows what he's doing.

Alright, this ATLAS stuff always puzzled me and I thought that one
has to have atlas3-base and atlas3-base-dev atlas3-headers
installed to use atlas3 during compilation.
I assumed that installing additionally (even afterwards)
atlas3-sse2 should give
optimal performance on the corresponding machine.
(Thinking about this, it is not clear why then atlas3-sse2-dev,
so the previous statement must be wrong ...)

OTOH, `apt-cache rdepends atlas3-base` shows a pretty long list,
including python2.3-scipy, python2.3-numeric-ext, python2.3-numarray-ext

OK, obviously I haven't understood the ATLAS setup of debian
and better shut up now and leave this for the experts .... ;-)

Tomorrow I will remove the atlas3-base stuff before building and
see how things work
(I don't need that urgently as building from source seems
easier, but the benefit of having proper debian packages
pays off very quickly in the longer run ...)

> > b) and the python2.3-dev *and* python2.4-dev dependency
> >
> This is a _build_ dependency. The source package builds python
> python2.3-numpy and python2.4-numpy, so it needs Python.h for both.

Alright, so no problem here - thanks for the clarification.

[...]

> > What about scipy: presently debian sarge comes with
> > scipy 0.3.2. Installing old-scipy and new-scipy side-by side
> > seems impossible (unless one does something like wxversion select
> > stuff...) - should the new scipy debs just replace the old ones?
> >
> Unless you do some apt-pinning, I think any new scipy (0.4.x) in any
> repository in your sources list will automatically override the old
> (0.3.x) simply via the versioning mechanisms of apt-get. I like the idea
> of a wxversion-alike, but I've shifted all my code to use numpy and the
> new scipy, so I don't have any motivation to do any implementation.

Also, it might not be completely trivial to set up and
there is still a lot of other stuff which has to be done ...

Best, Arnd

Re: [Numpy-discussion] .debs of numpy-0.9.8 available for Ubuntu Dapper

[Ed S. <sch...@ft...>]

On 08/06/2006, at 10:54 AM, Andrew Straw wrote:

> I've put together some .debs for numpy-0.9.8. There are binaries
> compiled for amd64 and i386 architectures of Ubuntu Dapper, and I
> suspect these will build from source for just about any Debian-based
> distro and architecture.
> ...

Great!

I posted an offer earlier this week to debian-science to help work on  
numpy packages (but got no response).  NumPy might be adopted much  
more rapidly once it has official packages in Debian and Ubuntu.  I'm  
glad you're in control of the situation; now I can now quietly  
withdraw my offer ;)

No, seriously ... I'd be happy to help out if I can :)

-- Ed

Re: [Numpy-discussion] .debs of numpy-0.9.8 available for Ubuntu Dapper

[Andrew S. <str...@as...>]

Andrew Straw wrote:

>I've put together some .debs for numpy-0.9.8. There are binaries 
>compiled for amd64 and i386 architectures of Ubuntu Dapper, and I 
>suspect these will build from source for just about any Debian-based 
>distro and architecture.
>  
>
As usually happens when I try to release packages in the middle of the 
night, the cold light of morning brings some glaring problems. The 
biggest one is that the .diff.gz that was generated wasn't showing the 
changes against numpy that I had to make. I'm surprised that my own 
tests with apt-get source showed that it still built from source.

I've uploaded a new version, 0.9.8-0ads2 (note the 2 at the end). You 
can check your installed version by doing the following:
dpkg-query -l *numpy*

Anyhow, here's the debian/changelog for 0.9.8-0ads2:

  * Fixed .orig.tar.gz so that .diff.gz includes modifications made
    to source.
  * Relax build-depend on setuptools to work with any version
  * Don't import setuptools in numpy.distutils.command.install unless
    it's already in sys.modules.

I would like to merge with the package in debian experimental by Jose 
Fonseca and Marco Presi, but their package uses a lot of makefile 
wizardry that bombs out on me without any apparently informative error 
message. (I will be the first to admit that I know very little about 
Makefiles.) On the other hand, the main advantage their package 
currently has is installation of manpages for f2py, installation of the 
existing free documentation, and tweaks to script (f2py) permissions and 
naming. The latter of these issues seems to be solved by the 
build-dependency on setuptools, which is smart about installing scripts 
with the right permissions and names (it appends "2.4" to the python2.4 
version of f2py, and so on).

There have been a couple of offers of help from Ed and Ryan. I think in 
the long run, the best thing to do would be to invest these efforts 
communicating with the debian developers and to get a more up-to-date 
version in their repository. (My repository will only ever be an 
unofficial repository with the primary purpose of serving our needs at 
work which hopefully overlaps substantially with usefulness to others.) 
This should have a trickle-down effect to mainline Ubuntu repository, 
also. I doubt that the debian developers will want to start their 
python-numpy package from scratch, so I can suggest trying to submit 
patches to their system. You can checkout their source at 
svn://svn.debian.org/deb-scipy . Unfortunately, that's about the only 
guidance I can provide, because, like I said above, I can't get their 
Makefile wizardry to work on a newer version of numpy.

Arnd, I would like to get to the bottom of these atlas issues myself, 
and I've followed a similar chain of logic as you. It's possible that 
the svd routine (dgesdd, IIRC) is somehow just a bad one to benchmark 
on. It is a real workhorse for me, and so it's really the one that 
counts for me. I'll put together a few timeit routines that test svd() 
and dot() and do some more experimentation, although I can't promise 
when. Let's keep everyone informed of any progress we make.

Cheers!
Andrew