numpy-discussion — Discussion list for all users of Numerical Python

[Numpy-discussion] 0-dimensional object arrays

[Ed S. <sch...@ft...>]

Hi all,

I've been attempting to use an object array to group together a
collection of different-length arrays or lists, and I'm getting
unexpected results.  Here's an example:

>>> import numpy
>>> a = numpy.array([[1,2],[1,2,3]],object)

I expected here that 'a' would be an array of dimension (2,), holding
two Python lists of integers.  Was I wrong to expect this?  It seems
instead that 'a' is a 0-dimensional array holding one object:

>>> a
array([[1, 2], [1, 2, 3]], dtype=object)
>>> a.shape
()
>>> type(a)
<type 'numpy.ndarray'>
>>> a[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
IndexError: 0-d arrays can't be indexed.

This behaviour seems less useful.  Would you agree this is a bug in the
array constructor?

I also can't explain this behaviour:

>>> [] + a
array([], dtype=object)
>>> a + []
array([], dtype=object)

I guess I would have expected these commands to be equivalent
array(a.view()+[]).

These results seem more sensible:

>>> a.transpose()
[[1, 2], [1, 2, 3]]
>>> a*2
[[1, 2], [1, 2, 3], [1, 2], [1, 2, 3]]

although these last two commands return Python lists, without the 0-d
array wrapper, which isn't consistent with other 0-d arrays.

On the whole, 0-d object arrays seem quite strange beasts.  Could
someone please enlighten me on why they deserve to exist? ;)  They seem
inconsistent with the new simplified interface to object array
elements.  Could we get rid of them entirely?!

-- Ed

[Numpy-discussion] Numeric/numpy difference

[Jens J. M. <je...@fy...>]

I just discovered (the hard way) that:

 >>> import numpy
 >>> a =3D numpy.empty((2, 3))
 >>> a[:] =3D [1, 2]
 >>> a
array([[1, 2, 1],
        [2, 1, 2]])

This is with revision 1984.  With Numeric I get "ValueError: matrices=20
are not aligned for copy", which I think is the correct answer.

Jens J=F8rgen

Re: [Numpy-discussion] Scalar array - a trap for the unwary

[Francesc A. <fa...@ca...>]

A Dilluns 23 Gener 2006 23:31, Colin J. Williams va escriure:
>  >>> a=3D _mu.array([1, 2], dtype=3D _nt.uint8)
>  >>> a+254
>
> array([255, 256], dtype=3Dint16)
>
> This does bring out a possible problem with the new type.  str of the
> old type returned the input name int8, float64 etc.  Is there some
> method of the new type which returns the name input, 'int8' in the above
> example?  In most cases, the mnemonic is more helpful than the letter
> coding.

Yes. a.dtype.name does the trick.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Numpy-discussion] Speed performance on array constant set

[Travis O. <oli...@ie...>]

Mark Heslep wrote:

> Travis Oliphant wrote:
>
>>> 1. numdata.h  example that works well for simple Cv structures  
>>> containing uniform types:  CvPoint2D32F  => struct { float x, float 
>>> y }.  Can I use a record array here, or is there some Numpy overhead 
>>> interlaced with fields?
>>
>>
>>
>> You can use a record array, but for uniform types I don't know what 
>> the advantage is (except for perhaps named-field slicing---which may 
>> actually be faster I'm not sure) over a x2 float array.
>
>
> So that I can step through the array record by record and not field by 
> field.

Right, but you could do that using steps of length 2 if you want to 
manage it yourself (or use a complex data type).    Again, I'm not sure 
of the performance implications at this point as I haven't done enough 
tests.

>
>>> 2. numarray.h  Attempt to replace the main Cv Image structures 
>>> CvMat, IplImage.  Needs work. Some success but there's segfault or 
>>> two in there somewhere.
>>
>>
>> These can be ported fairly easily, I think (actually, the old Numeric 
>> typemaps would still work --- and work with Numarray), so the basic 
>> Numeric C-API still presents itself as the simplest way to support 
>> all the array packages.
>
>
> Well good to know.  Ive been proceeding directly from the NumArray 
> Users Manual 1.5 by Greenfield/Miller et al that describes the NA High 
> Level API as the fastest of the APIs available to NA.  I thought that 
> NA also took steps to insure that unnecessary mem copies were not 
> made, unlike Numeric?


Numeric doesn't make unnecessary copies either.  It depends on what you 
mean by necessary.  I think they may be referring to the fact that most 
people used ContiguousFromObject which always gave you contiguous memory 
from an array (copying if necessary), but if you learn how to access 
strided memory then FromObject was just fine (which would never copy). 

There are still some algorithms that don't deal with strided memory and 
so will make a copy.  Some of these can be changed, others would be more 
difficult.   It really becomes a question of cache-misses versus memory 
copy time, which I'm not sure how to optimize generally except by 
per-system experimentation like ATLAS does.

-Travis

Re: [Numpy-discussion] Speed performance on array constant set

[Mark H. <ma...@mi...>]

Travis Oliphant wrote:

>> 1. numdata.h  example that works well for simple Cv structures  
>> containing uniform types:  CvPoint2D32F  => struct { float x, float y 
>> }.  Can I use a record array here, or is there some Numpy overhead 
>> interlaced with fields?
>
>
> You can use a record array, but for uniform types I don't know what 
> the advantage is (except for perhaps named-field slicing---which may 
> actually be faster I'm not sure) over a x2 float array.

So that I can step through the array record by record and not field by 
field.

>> 2. numarray.h  Attempt to replace the main Cv Image structures CvMat, 
>> IplImage.  Needs work. Some success but there's segfault or two in 
>> there somewhere.
>
> These can be ported fairly easily, I think (actually, the old Numeric 
> typemaps would still work --- and work with Numarray), so the basic 
> Numeric C-API still presents itself as the simplest way to support all 
> the array packages.

Well good to know.  Ive been proceeding directly from the NumArray Users 
Manual 1.5 by Greenfield/Miller et al that describes the NA High Level 
API as the fastest of the APIs available to NA.  I thought that NA also 
took steps to insure that unnecessary mem copies were not made, unlike 
Numeric?

Mark

Re: [Numpy-discussion] Eigenvalues hangs

[Sasha <nd...@ma...>]

Sorry for an empty message.  I lost connection and a half a page
message with it :-(

Here is a brief summary:

-ffloat-store may be expensive:

"""
Reading the gcc documentation, one finds that the -ffloat-store is an
expensive hack to attempt to avoid the excess precision in the floating
pointer registers on Intel FPUs.  Based on recent experience with
DONLP2, a noteworthy nonlinear solver, using -ffloat-store everywhere is
an unnecessarily costly workaround.
"""
<http://www.octave.org/octave-lists/archive/octave-maintainers.2004/msg0082=
5.html>

If it has to be enabled, it should be limited to dlapack_lite .

Red Hat patched its own lapack by disabling optimization for a few routines=
:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=3D138683

-- sasha





On 1/23/06, Sasha <nd...@ma...> wrote:
> On 1/23/06, Sasha <nd...@ma...> wrote:
> > I think this issue was altimately resolved as a lapack rather than gcc
> > problem and -ffloat-store is not the right work-around.  I don't have
> > the right references right now, so unless someone steps in I will
> > follow up when I get home this evening.
> >
> > -- sasha
> >
> > On 1/23/06, S=E9bastien Boisg=E9rault <boi...@is...> wrote:
> > >
> > > Hi all,
> > >
> > > Robert Kern suggested to transfer a discussion that originated
> > > on comp.lang.python to this mailing-list.
> > >
> > > My problem is that on my platform the numpy 'eigenvalues' method
> > > hangs forever.
> > >
> > > Some platform info:
> > >     * numpy 0.9.4 (from the tar.gz)
> > >     * Linux Mandriva (gcc 4.0.1).
> > >     * no optimized BLAS version used.
> > >
> > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and
> > > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS.
> > >
> > > Could anybody tell me how I should modify the numpy sources to test
> > > this workaround ?
> > >
> > > Regards,
> > >
> > > SB
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.net email is sponsored by: Splunk Inc. Do you grep through lo=
g files
> > > for problems?  Stop!  Download the new AJAX search engine that makes
> > > searching your log files as easy as surfing the  web.  DOWNLOAD SPLUN=
K!
> > > http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D103432&bid=3D230486&d=
at=3D121642
> > > _______________________________________________
> > > Numpy-discussion mailing list
> > > Num...@li...
> > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> > >
> >
>

Re: [Numpy-discussion] Eigenvalues hangs

[Sasha <nd...@ma...>]

On 1/23/06, Sasha <nd...@ma...> wrote:
> I think this issue was altimately resolved as a lapack rather than gcc
> problem and -ffloat-store is not the right work-around.  I don't have
> the right references right now, so unless someone steps in I will
> follow up when I get home this evening.
>
> -- sasha
>
> On 1/23/06, S=E9bastien Boisg=E9rault <boi...@is...> wrote:
> >
> > Hi all,
> >
> > Robert Kern suggested to transfer a discussion that originated
> > on comp.lang.python to this mailing-list.
> >
> > My problem is that on my platform the numpy 'eigenvalues' method
> > hangs forever.
> >
> > Some platform info:
> >     * numpy 0.9.4 (from the tar.gz)
> >     * Linux Mandriva (gcc 4.0.1).
> >     * no optimized BLAS version used.
> >
> > Greg Landrum pointed out that it may be a gcc 4.0 related problem and
> > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS.
> >
> > Could anybody tell me how I should modify the numpy sources to test
> > this workaround ?
> >
> > Regards,
> >
> > SB
> >
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc. Do you grep through log =
files
> > for problems?  Stop!  Download the new AJAX search engine that makes
> > searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> > http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D103432&bid=3D230486&dat=
=3D121642
> > _______________________________________________
> > Numpy-discussion mailing list
> > Num...@li...
> > https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> >
>

Re: [Numpy-discussion] Speed performance on array constant set

[Travis O. <oli...@ee...>]

Mark Heslep wrote:

> Christopher Barker wrote:
>
>> Mark Heslep wrote:
>>
>>> Ive got some NumArray C API investment in the typemaps.
>>
>>
>>
>> Are these SWIG typemaps? If so I think there are a bunch of us that 
>> wold like to see them, and maybe we can all work together to port 
>> them to NumPy.
>>
>> -Chris
>>
> Yes they are SWIG typemaps for OpenCv* data structures, based on prior 
> Numeric art I'd found and ported to Numarray.
> Attached.
>
> 1. numdata.h  example that works well for simple Cv structures  
> containing uniform types:  CvPoint2D32F  => struct { float x, float y 
> }.  Can I use a record array here, or is there some Numpy overhead 
> interlaced with fields?

You can use a record array, but for uniform types I don't know what the 
advantage is (except for perhaps named-field slicing---which may 
actually be faster I'm not sure) over a x2 float array. 

> 2. numarray.h  Attempt to replace the main Cv Image structures CvMat, 
> IplImage.  Needs work. Some success but there's segfault or two in 
> there somewhere.
>
These can be ported fairly easily, I think (actually, the old Numeric 
typemaps would still work --- and work with Numarray), so the basic 
Numeric C-API still presents itself as the simplest way to support all 
the array packages.

-Travis

Re: [Numpy-discussion] Speed performance on array constant set

[Mark H. <ma...@mi...>]

Christopher Barker wrote:

> Mark Heslep wrote:
>
>> Ive got some NumArray C API investment in the typemaps.
>
>
> Are these SWIG typemaps? If so I think there are a bunch of us that 
> wold like to see them, and maybe we can all work together to port them 
> to NumPy.
>
> -Chris
>
Yes they are SWIG typemaps for OpenCv* data structures, based on prior 
Numeric art I'd found and ported to Numarray.
Attached.

1. numdata.h  example that works well for simple Cv structures  
containing uniform types:  CvPoint2D32F  => struct { float x, float y 
}.  Can I use a record array here, or is there some Numpy overhead 
interlaced with fields?
2. numarray.h  Attempt to replace the main Cv Image structures CvMat, 
IplImage.  Needs work. Some success but there's segfault or two in there 
somewhere.
 
 *Details for Cv use:  I drop them in the  ../interfaces/swig/python path
 for opencv and <include> them  in cv.i
 
 Regards,
 Mark

Re: [Numpy-discussion] include file location and .eggs

[Christopher B. <Chr...@no...>]

kon...@la... wrote:
> Distutils doesn't provide a perfect solution, but if everyone just  lets 
> distutils put header files into the default location, and all  packages 
> get installed without an explicit prefix, then it will just  work. 
> Considering that I get next to no support questions about such  
> installation issues, I consider the current situation very acceptable.

Except that there is no support whatsoever for multiple versions. It 
seems your complicated scenario was due to multiple versions. In fact, 
the one problem I've had with Scientific in the past was trying to use a 
binary that had been compiled against a different version of Numeric 
than I was using.

If your user is compiling everything, then there is no problem. If your 
user is using binaries, they they need to be labeled with versions. If 
they use the wrong one with eggs, then there is at least some hope that 
they'll get a meaningful error message.

If they are compiling MMTK against a Scientific and a Numpy that they 
didn't compile, then it's going to get ugly -- but I don't think it's 
any uglier than it is now.

As for improving distutils in the next go around -- I'm all for it, but 
why not have 2.5 include setuptools instead -- it strikes me as a next 
generation distutils.

I do agree that setuptools should handle the header file placement and 
finding for you, rather than each package having to do it on it's own.


and Andrew, thanks for all the info on setuptools and eggs. It's been 
very informative!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Scalar array - a trap for the unwary

[Colin J. W. <cj...@sy...>]

Travis Oliphant wrote:

> Colin J. Williams wrote:
>
>> In exploring these matters, it seems that numpy does not broadcast 
>> scalar operations on arrays, as numarray does.  Is this intended?  
>> Please see the script below and the result under that.
>
>
> I don't know what is going on with your installation, because I don't 
> get that error.  Your script works fine for me.

I've just tried this with version 0.9.4 and all works well.
 >>> a= _mu.array([1, 2], dtype= _nt.uint8)
 >>> a+254
array([255, 256], dtype=int16)

This does bring out a possible problem with the new type.  str of the 
old type returned the input name int8, float64 etc.  Is there some 
method of the new type which returns the name input, 'int8' in the above 
example?  In most cases, the mnemonic is more helpful than the letter 
coding.

>
> I wouldn't advise extracting array directly from multiarray as you are 
> doing, because exactly where array lives is an implementation detail 
> that could change in the future.

I accept this risk for now.  Partly, this was done to simplify the 
namespace and partly to locate a bug concerning the interaction of a 
Windows debugger and the umath module.

>
> -Travis
>
Colin W.

Re: [Numpy-discussion] include file location and .eggs

[<kon...@la...>]

On 23.01.2006, at 19:17, Pearu Peterson wrote:

> I don't know about setuptools but distutils cannot solve this  
> problem either without some extra code in MMTK setup.py file and a  
> possibility to determine which choice was used to build  
> ScientificPython. So, it's not really a distutils nor numpy include  
> file location issue.

Distutils doesn't provide a perfect solution, but if everyone just  
lets distutils put header files into the default location, and all  
packages get installed without an explicit prefix, then it will just  
work. Considering that I get next to no support questions about such  
installation issues, I consider the current situation very acceptable.

>> And how is it supposed to know where each of these packages stores  
>> its header files?
>
> Here's an answer to this question, namely, when numpy is installed,  
> then one can use the following code in setup.py to support  
> different array backends:

That's for numpy. How about Numeric and numarray? And how about  
whatever other packages need header files? If everyone makes up his  
own conventions, we end up with a big mess.


Another idea: considering that setup.py needs to be changed anyway to  
make eggs (because setup must be imported from somewhere else), why  
not put some flag "make_egg" at the beginning of numpy's setup.py,  
and depending on the value of the flag either use setuptools.setup  
and header files in the data section, or distutils.core.setup and  
header files in the standard location? That should suit everyone, right?

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: kh...@ce...
------------------------------------------------------------------------ 
-------

Re: [Numpy-discussion] Eigenvalues hangs

[Sasha <nd...@ma...>]

I think this issue was altimately resolved as a lapack rather than gcc
problem and -ffloat-store is not the right work-around.  I don't have
the right references right now, so unless someone steps in I will
follow up when I get home this evening.

-- sasha

On 1/23/06, S=E9bastien Boisg=E9rault <boi...@is...> wrote:
>
> Hi all,
>
> Robert Kern suggested to transfer a discussion that originated
> on comp.lang.python to this mailing-list.
>
> My problem is that on my platform the numpy 'eigenvalues' method
> hangs forever.
>
> Some platform info:
>     * numpy 0.9.4 (from the tar.gz)
>     * Linux Mandriva (gcc 4.0.1).
>     * no optimized BLAS version used.
>
> Greg Landrum pointed out that it may be a gcc 4.0 related problem and
> proposed a workaround -- to add the option '-ffloat-store' to CFLAGS.
>
> Could anybody tell me how I should modify the numpy sources to test
> this workaround ?
>
> Regards,
>
> SB
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log fi=
les
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D103432&bid=3D230486&dat=
=3D121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>

Re: [Numpy-discussion] Eigenvalues hangs

[<boi...@is...>]

David M. Cooke a =C3=A9crit :

>S=C3=A9bastien Boisg=C3=A9rault <boi...@is...> writes:
>
> =20
>
>>Hi all,
>>
>>Robert Kern suggested to transfer a discussion that originated
>>on comp.lang.python to this mailing-list.
>>
>>My problem is that on my platform the numpy 'eigenvalues' method
>>hangs forever.
>>
>>Some platform info:
>>   * numpy 0.9.4 (from the tar.gz)
>>   * Linux Mandriva (gcc 4.0.1).
>>   * no optimized BLAS version used.
>>
>>Greg Landrum pointed out that it may be a gcc 4.0 related problem and
>>proposed a workaround -- to add the option '-ffloat-store' to CFLAGS.
>>
>>Could anybody tell me how I should modify the numpy sources to test
>>this workaround ?
>>   =20
>>
>
>You should just be able to do the setup like this:
>
>$ CFLAGS=3D'-ffloat-store' python setup.py build
>
> =20
>
Great ! It solves the problem :)

Thanks a lot for the help !

SB

Re: [Numpy-discussion] Eigenvalues hangs

[<co...@ph...>]

S=C3=A9bastien Boisg=C3=A9rault <boi...@is...> writes:

> Hi all,
>
> Robert Kern suggested to transfer a discussion that originated
> on comp.lang.python to this mailing-list.
>
> My problem is that on my platform the numpy 'eigenvalues' method
> hangs forever.
>
> Some platform info:
>    * numpy 0.9.4 (from the tar.gz)
>    * Linux Mandriva (gcc 4.0.1).
>    * no optimized BLAS version used.
>
> Greg Landrum pointed out that it may be a gcc 4.0 related problem and
> proposed a workaround -- to add the option '-ffloat-store' to CFLAGS.
>
> Could anybody tell me how I should modify the numpy sources to test
> this workaround ?

You should just be able to do the setup like this:

$ CFLAGS=3D'-ffloat-store' python setup.py build

--=20
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

[Numpy-discussion] Eigenvalues hangs

[<boi...@is...>]

Hi all,

Robert Kern suggested to transfer a discussion that originated
on comp.lang.python to this mailing-list.

My problem is that on my platform the numpy 'eigenvalues' method
hangs forever.

Some platform info:
    * numpy 0.9.4 (from the tar.gz)
    * Linux Mandriva (gcc 4.0.1).
    * no optimized BLAS version used.

Greg Landrum pointed out that it may be a gcc 4.0 related problem and
proposed a workaround -- to add the option '-ffloat-store' to CFLAGS.

Could anybody tell me how I should modify the numpy sources to test
this workaround ?

Regards,

SB

Re: [Numpy-discussion] include file location and .eggs

[Pearu P. <pe...@sc...>]

On Mon, 23 Jan 2006, kon...@la... wrote:

> Let's go back to my initial scenario, and let's even forget about multiple 
> versions. I have ScientificPython, which can be compiled for use with either 
> Numeric or numarray, and some day NumPy will be a third choice. Then I 
> install MMTK, which depends on ScientificPython. How is the MMTK installation 
> script supposed to figure out if ScientificPython was compiled with Numeric, 
> numarray, or NumPy?

I don't know about setuptools but distutils cannot solve this problem 
either without some extra code in MMTK setup.py file and a possibility to 
determine which choice was used to build ScientificPython. So, it's not 
really a distutils nor numpy include file location issue.

> And how is it supposed to know where each of these packages stores its header files?

Here's an answer to this question, namely, when numpy is installed, 
then one can use the following code in setup.py to support different array 
backends:

   from numpy.distutils.system_info import get_info
   arr_info = get_info('numpy') or get_info('numarray') or get_info('Numeric')

   # For explanation, arr_info is one of the following dictionaries:
   #  {'define_macros':[('NUMPY',None)], 'include_dirs':['/path/to/numpy/header/dir']}
   #  {'define_macros':[('NUMARRAY',None)], 'include_dirs':['/path/to/numarray/header/dir']}
   #  {'define_macros':[('NUMERIC',None)], 'include_dirs':['/path/to/Numeric/header/dir']}
   #  {} # --- no array backends installed
   # If one has all three array backend packages installed and wishes to
   # use, say, numarray, then to disable detecting numpy, one should
   # define environment variable NUMPY=None.

   ...

   setup(...,
      ext_modules = [Extension(..., **arr_info),..])

Pearu

Re: [Numpy-discussion] include file location and .eggs

[<kon...@la...>]

On Jan 23, 2006, at 19:03, Andrew Straw wrote:

> The problem is backwards compatibility. Nothing like this is =20
> implemented
> in plain distutils, which is why the packages will have to support it
> themselves. This way they can keep working in both old distutils-based
> environments and new ones, including with setuptools. The current
> solution _works_ on yesteryear's Python 2.3. In my opinion, relying on

Well, I'd argue it doesn't work at all, because it doesn't scale to =20
complex dependency trees.

> Distutils never handled the case when, for example you installed =20
> Numeric
> with '/usr/local/bin/python2.3 setup.py install
> --prefix=3D/home/astraw/py23'. In this case, sure, the package got
> installed where I wanted (in my $PYTHONPATH) in
> /home/astraw/py23/lib/python2.3/site-packages/Numeric, but the headers
> were in /home/astraw/py23/include/python2.3/Numeric. Distutils would

True. It was always true that if you install anywhere but in the =20
default location, you are on your own. But if you do keep the default =20=

location (as most users do), everything works fine.

> never find those headers again. This is exactly the same problem, and
> it's one that distutils never solved, and I don't think we should hold
> our breath waiting for Python 2.5's distutils to solve it. For one

Why not contribute to solving it?

> thing, it's already solved at the package level a la numpy. For =20
> another,
> given that it's already solved, who's going to bother attempting to =20=

> make

I don't think it is solved at all.

> Why is "import numpy; include_dirs=3Dnumpy.get_numpy_include()" in
> setup.py unpalatable for you? That's all that is required in setup.py.

Let's go back to my initial scenario, and let's even forget about =20
multiple versions. I have ScientificPython, which can be compiled for =20=

use with either Numeric or numarray, and some day NumPy will be a =20
third choice. Then I install MMTK, which depends on ScientificPython. =20=

How is the MMTK installation script supposed to figure out if =20
ScientificPython was compiled with Numeric, numarray, or NumPy? And =20
how is it supposed to know where each of these packages stores its =20
header files?

I don't mind adding some lines to setup.py for each package, that's a =20=

small price to pay. But I do mind having to worry about indirect =20
dependencies.

> to argue this is more pain than you're willing to deal with, I think
> that, to be fair, you need to give a counter-example whereby you =20
> explain
> how you handle all those versions of Numeric and Scientific installed
> simultaneously and how you select among them. I'll admit that in the

I don't, except on my development machine, where I used symbolic =20
links that are changed using scripts. That's a bit of work to set up, =20=

but I have only two packages of which I use multiple versions, so =20
doing it once is acceptable for me.

In fact, my main worry is not my own machine, nor the machine of any =20
competent Python programmer. My worry is the less competent user who =20
can't get installation of my packages to work and turns to me for =20
help. Tech support takes more of my time than development these days. =20=

A fragile dependency system such as the one you propose will only =20
create more support questions on my side - so don't expect me to =20
support it.

> Like I said above, distutils has no such equivalent mechanism, and =20
> this
> _allows_ something that simply wasn't possible before and is arguably
> broken in distutils.

I am all for fixing distutils.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L=E9on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: kh...@ce...
---------------------------------------------------------------------

Re: [Numpy-discussion] include file location and .eggs

[Andrew S. <str...@as...>]

kon...@la... wrote:

> On Jan 23, 2006, at 18:07, Andrew Straw wrote:
>
>> Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5.
>> You'd make sure the call to setup() (which is now imported from
>> setuptools, not distutils.core) included the following:
>
>
> ...
>
> OK, that looks sophisticated enough to me :-)
>
> Since setuptools keeps track of dependencies, it should be able to 
> provide on demand a list of all include file directories for all 
> packages on which the current build procedure depends, directly or 
> indirectly. To stay within my example, a package that depends on 
> ScientificPython, which in turn depends on Numeric, should be able to 
> get, with a single call, the include directories for *both* 
> ScientificPython and Numeric.
>
> Assuming such a mechanism exists, and is also implemented in plain 
> distutils (returning just the standard installation paths), then 
> personally I wouldn't care where header files are, because there is a 
> straightforward way to obtain that information.

The problem is backwards compatibility. Nothing like this is implemented
in plain distutils, which is why the packages will have to support it
themselves. This way they can keep working in both old distutils-based
environments and new ones, including with setuptools. The current
solution _works_ on yesteryear's Python 2.3. In my opinion, relying on
distutils to manage header file locations was a mistake, but that's easy
to say in hindsight, when it's clear that, as useful as distutils is, it
has some deficiencies.

Distutils never handled the case when, for example you installed Numeric
with '/usr/local/bin/python2.3 setup.py install
--prefix=/home/astraw/py23'. In this case, sure, the package got
installed where I wanted (in my $PYTHONPATH) in
/home/astraw/py23/lib/python2.3/site-packages/Numeric, but the headers
were in /home/astraw/py23/include/python2.3/Numeric. Distutils would
never find those headers again. This is exactly the same problem, and
it's one that distutils never solved, and I don't think we should hold
our breath waiting for Python 2.5's distutils to solve it. For one
thing, it's already solved at the package level a la numpy. For another,
given that it's already solved, who's going to bother attempting to make
a patch (which can't be relied on by older pythons) for Python 2.5,
which may not even be accepted?

>
> However, that is not what is currently being discussed for numpy, 
> unless I missed something. What is being discussed is an arrangement 
> that requires the setup.py of every package to reconstruct the 
> dependency tree and figure out where each package in the dependency 
> tree keeps its header files.

Why is "import numpy; include_dirs=numpy.get_numpy_include()" in
setup.py unpalatable for you? That's all that is required in setup.py.
To me, this is a small price to pay for the benefits gained. If you want
to argue this is more pain than you're willing to deal with, I think
that, to be fair, you need to give a counter-example whereby you explain
how you handle all those versions of Numeric and Scientific installed
simultaneously and how you select among them. I'll admit that in the
case you gave, of  2 versions of Scientific and 2 versions of Numeric
compiled into all 4 possible combinations, you'd also need an "import
pkg_resources; pkg_resources.require('Numeric==x.y')" prior to the
"import Numeric". I see the case you gave as a rather extreme case --
normally I would like, for example, a single version of (old) scipy
compiled against old Numeric and a single new version of scipy compiled
against numpy. (If it's not already working, I'm almost willing to
backport egg support to old scipy to do just this...)

Like I said above, distutils has no such equivalent mechanism, and this
_allows_ something that simply wasn't possible before and is arguably
broken in distutils.

>
>
> There is something else that I don't understand. If egg installation 
> uses setup from setuptools instead of distutils, it should be trivial 
> for the setup function from setuptools to implement install_headers 
> in a way that suits its need. So why request that numpy (and other 
> packages) adapt to setuptools, creating compatibility issues with 
> standard distutils, rather than modify setuptools and have it work 
> automatically for all packages?

That's an interesting idea. The short answer is that setuptools doesn't
do it currently. Perhaps it could. I'm not going to implement it,
though. It also doesn't solve the problem I gave above with plain
distutils, which I think is a real problem that numpy solves in a
backwards compatible way.

Re: [Numpy-discussion] include file location and .eggs

[<kon...@la...>]

On Jan 23, 2006, at 18:07, Andrew Straw wrote:

> Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5.
> You'd make sure the call to setup() (which is now imported from
> setuptools, not distutils.core) included the following:

...

OK, that looks sophisticated enough to me :-)

Since setuptools keeps track of dependencies, it should be able to =20
provide on demand a list of all include file directories for all =20
packages on which the current build procedure depends, directly or =20
indirectly. To stay within my example, a package that depends on =20
ScientificPython, which in turn depends on Numeric, should be able to =20=

get, with a single call, the include directories for *both* =20
ScientificPython and Numeric.

Assuming such a mechanism exists, and is also implemented in plain =20
distutils (returning just the standard installation paths), then =20
personally I wouldn't care where header files are, because there is a =20=

straightforward way to obtain that information.

However, that is not what is currently being discussed for numpy, =20
unless I missed something. What is being discussed is an arrangement =20
that requires the setup.py of every package to reconstruct the =20
dependency tree and figure out where each package in the dependency =20
tree keeps its header files.


There is something else that I don't understand. If egg installation =20
uses setup from setuptools instead of distutils, it should be trivial =20=

for the setup function from setuptools to implement install_headers =20
in a way that suits its need. So why request that numpy (and other =20
packages) adapt to setuptools, creating compatibility issues with =20
standard distutils, rather than modify setuptools and have it work =20
automatically for all packages?

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L=E9on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: kh...@ce...
---------------------------------------------------------------------

Re: [Numpy-discussion] include file location and .eggs

[Andrew S. <str...@as...>]

kon...@la... wrote:

> On Jan 19, 2006, at 18:31, Andrew Straw wrote:


>> = Why can't eggs use traditional include locations? =
>> Multiple version support. Using eggs, it is possible to have multiple
>> versions of the same distribution, such as numpy, installed within a
>> single Python installation. (I use "distribution" to mean  collection of
>
>
> How is multiple version support in eggs supposed to work in the case 
> of packages that contain C modules?

Eggs will allow you to do it, but you'll have to specify the exact
dependencies in the setup.py file.

>
> Let me give a concrete example: Numeric, ScientificPython, and MMTK. 
> Numeric provides a C API through its header files. ScientificPython 
> and MMTK both contain C modules that depend on that C API. 
> ScientificPython also provides a C API for two of its modules, which 
> in turn MMTK depends on.

Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5.
You'd make sure the call to setup() (which is now imported from
setuptools, not distutils.core) included the following:

setup( name='Scientific',
    version='2.4.2',
    ext_modules=stuff_that_depends_on_Numeric_23_5,
    install_requires='Numeric==23.5')

Clearly, you could do this for all versions of Numeric and Scientific
you want to run together. Obviously, it could be done in a more
automated way (by querying what version of Numeric is being used to
build). To have two versions of Scientific 2.4.2 installed, however,
they would have to have different names, so perhaps '2.4.2_Numeric23.5'
would work well for the example above. Now, you can just do your imports
as normal, which will result in the the highest version number
available. To specify versions, use 'import pkg_resources;
pkg_resources.require('Numeric==23.5'); import Numeric'.

An exception is raised if you do 'import Numeric' and it imports
Numeric==24.2 and then you later try and import Scientific with only
version 2.4.2_Numeric23.5 available. (At least, if an exception isn't
raised, I'm 99% sure this would be considered a bug and would be fixed.)
So, the import order is important -- it's probably best to import
Scientific first in this example so that the correct version of Numeric
can be selected behind the scenes without requiring you to write any
pkg_resources.require() calls.

N.B. For setuptools to get a package's (e.g. Numeric's) version number,
Numeric must be built with setuptools -- not necessarily as an egg, but
at least with the egg metadata available. If you don't build as an egg,
though, you can't have more than one version installed. FYI, to build a
conventional (non-egg) with setuptools so that it includes this metadata
from an unmodified package that doesn't itself use setuptools, use the
following
python -c "import setuptools; execfile('setup.py')" install
--single-version-externally-managed

>
> Suppose I would like to have both Numeric 23.5 and Numeric 24.2 on my 
> machine, and also Scientific 2.4.2 and Scientific 2.5.1. I would then 
> need four sets of the binary modules of Scientific:
>
> 1) 2.4.2 for Numeric 23.5
> 2) 2.4.2 for Numeric 24.2
> 3) 2.5.1 for Numeric 23.5
> 4) 2.5.1 for Numeric 24.2

See above.

>
> I would also need four sets of the binary modules of MMTK, assuming 
> that I am happy with just one version of MMTK.

The above scales, so I don't think any new info is needed here.

>
> Unless some mechanism (which would have to be part of egg 
> construction or egg installation) makes sure that the correct 
> binaries are used for the particular combination of versions that I 
> wish to use, I will end up having lots of mysterious crashes 
> resulting from incompatible binary versions.

That's exactly the point -- eggs do ensure the correct binary version is
used. To come full circle to the question of header file location, this
is exactly why eggs need to have multiple versions of the header files
available, each version located within their respective egg. And this is
also why keeping the header files in
/usr/local/include/python2.4/Numeric/arrayobject.h is incompatible with
one of the best features of eggs.

>
> If the egg mechanism does take care of such dependencies, then it 
> could easily (and thus should) take care of selecting the right 
> combination of header files. If it doesn't, then it should not be 
> used at all for packages containing C or Pyrex modules, and packages 
> like numpy should better not accomodate to eggs because in the long 
> run that will only cause support headaches.

So how, about the converse? :) If setuptools does take care of
dependencies (and it does), should packages like numpy definitely
accommodate eggs? I certainly think so! :)

Andrew

Re: [Numpy-discussion] Speed performance on array constant set

[Travis O. <oli...@ie...>]

Mark Heslep wrote:

> Travis Oliphant wrote:
>
>> First of all,  try using NumPy instead of Numarray:    import numpy 
>> as na
>>
> Ive got some NumArray C API investment in the typemaps.  Is the C high 
> level API the same, ie. NA_Input, etc?  I havnt found the C API docs 
> on the web site.


The high-level C-API is the same as the old Numeric C-API.  The new 
numarray C-API is not supported at this point.  However, I would like to 
see a numarray compatibility module written so that it is easy to port 
numarray-based C-routines (simply by changing the header). 

I've started such a thing in scipy/Lib/sandbox/nd_image area.  Look at 
these files for guidance:

http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/nd_image/Src/numcompat.h
http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/nd_image/Src/numcompat.c

I'm thinking that a module could be written that contains the entire 
numarray C-API.  


-Travis

Re: [Numpy-discussion] Speed performance on array constant set

[Mark H. <ma...@mi...>]

Travis Oliphant wrote:

> First of all,  try using NumPy instead of Numarray:    import numpy as na
>
Ive got some NumArray C API investment in the typemaps.  Is the C high 
level API the same, ie. NA_Input, etc?  I havnt found the C API docs on 
the web site.

> Second:   use math (i.e.   a += 7)
>
> Third:   There are specialized fill routines .fill()  in numpy and a 
> simliar routine in Numarray that can be faster then indexing. 

Im happy to help out on the optimization of .fill if needed

Mark

Re: [Numpy-discussion] Scalar array - a trap for the unwary

[Travis O. <oli...@ie...>]

Colin J. Williams wrote:

> In exploring these matters, it seems that numpy does not broadcast 
> scalar operations on arrays, as numarray does.  Is this intended?  
> Please see the script below and the result under that.

I don't know what is going on with your installation, because I don't 
get that error.  Your script works fine for me.

I wouldn't advise extracting array directly from multiarray as you are 
doing, because exactly where array lives is an implementation detail 
that could change in the future.

-Travis

Re: [Numpy-discussion] include file location and .eggs

[<kon...@la...>]

On Jan 19, 2006, at 18:31, Andrew Straw wrote:

> For me, and I suspect for most others in favor of the new way, the
> answer boils down to using eggs with out-of-the box numpy. Eggs are =20=

> the
...

Thanks for explaining this issue in some detail!

> =3D Why can't eggs use traditional include locations? =3D
> Multiple version support. Using eggs, it is possible to have multiple
> versions of the same distribution, such as numpy, installed within a
> single Python installation. (I use "distribution" to mean =20
> collection of

How is multiple version support in eggs supposed to work in the case =20
of packages that contain C modules?

Let me give a concrete example: Numeric, ScientificPython, and MMTK. =20
Numeric provides a C API through its header files. ScientificPython =20
and MMTK both contain C modules that depend on that C API. =20
ScientificPython also provides a C API for two of its modules, which =20
in turn MMTK depends on.

Suppose I would like to have both Numeric 23.5 and Numeric 24.2 on my =20=

machine, and also Scientific 2.4.2 and Scientific 2.5.1. I would then =20=

need four sets of the binary modules of Scientific:

1) 2.4.2 for Numeric 23.5
2) 2.4.2 for Numeric 24.2
3) 2.5.1 for Numeric 23.5
4) 2.5.1 for Numeric 24.2

I would also need four sets of the binary modules of MMTK, assuming =20
that I am happy with just one version of MMTK.

Unless some mechanism (which would have to be part of egg =20
construction or egg installation) makes sure that the correct =20
binaries are used for the particular combination of versions that I =20
wish to use, I will end up having lots of mysterious crashes =20
resulting from incompatible binary versions.

If the egg mechanism does take care of such dependencies, then it =20
could easily (and thus should) take care of selecting the right =20
combination of header files. If it doesn't, then it should not be =20
used at all for packages containing C or Pyrex modules, and packages =20
like numpy should better not accomodate to eggs because in the long =20
run that will only cause support headaches.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L=E9on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: kh...@ce...
---------------------------------------------------------------------