numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] In-place operations

[Francesc A. <fa...@ca...>]

El dt 12 de 09 del 2006 a les 13:28 -0600, en/na Travis Oliphant va
escriure:
> >[BTW, numpy.empty seems twice as slower in my machine. Why?
> > =20
> >
> >>>>Timer("a=3Dnumpy.empty(10000,dtype=3Dnumpy.complex128)", "import
> >>>>       =20
> >>>>
> >numpy").repeat(3,10000)
> >[0.37033700942993164, 0.31780219078063965, 0.31607294082641602]
> >]
> > =20
> >
> Now, you are creating an empty array with 10000 elements in it.=20

Ups, my bad. So, here are the correct times for array creation:

>>> Timer("a=3Dnumpy.empty(10,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.083303928375244141, 0.080381870269775391, 0.077172040939331055]
>>> Timer("a=3Dnumpy.empty(100,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.086454868316650391, 0.084085941314697266, 0.083555936813354492]
>>> Timer("a=3Dnumpy.empty(1000,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.084996223449707031, 0.082299947738647461, 0.081347942352294922]
>>> Timer("a=3Dnumpy.empty(10000,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.31068897247314453, 0.30376386642456055, 0.30176281929016113]
>>> Timer("a=3Dnumpy.empty(100000,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.42552995681762695, 0.36864185333251953, 0.36870002746582031]
>>> Timer("a=3Dnumpy.empty(1000000,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.48045611381530762, 0.41251182556152344, 0.40645909309387207]

So, it seems that there are a certain time dependency with size

array of 10 elements       -->  7.7 us
array of 100 elements      -->  8.4 us
array of 1000 elements     -->  8.1 us
array of 10000 elements    --> 30.2 us
array of 100000 elements   --> 36.9 us
array of 1000000 elements  --> 40.6 us

Well, it seems that malloc actually takes more time when asking for more
space. However, this can't be the reason why Pierre is seeing that:

a =3D numpy.exp(a)  [1]

is slower than

numpy.exp(a,out=3Da)  [2]

as I'd say that this increment in time is negligible compared with
processing times of those big arrays. In fact, here are my times:

>>> Timer("a =3D numpy.exp(a)", "import numpy;a =3D
numpy.random.rand(2048,2048) + 1j *
numpy.random.rand(2048,2048)").repeat(3,1)
[2.5527338981628418, 2.5427830219268799, 2.5074479579925537]
>>> Timer("numpy.exp(a,out=3Da)", "import numpy;a =3D
numpy.random.rand(2048,2048) + 1j *
numpy.random.rand(2048,2048)").repeat(3,1)
[2.5298278331756592, 2.5082788467407227, 2.5222280025482178]

So, both times are comparable.

Perhaps what Pierre is seeing is that he is approaching the limits of
memory in his system and because [1] takes more memory than [2] (two
objects in memory instead of one) perhaps the former is causing the OS
to start swapping. However a quick look with top at the processes, says
that both [1] and [2] takes similar amounts of memory (~ 170 MB peak)
and, as arrays take 64 MB each, in both cases the used memory seems
higher than the required at first sight. Mmmm, the only explanation is
that the exp() ufunc does require temporaries, although this is a bit
strange as exp() works element wise. I recognize that I'm a bit lost
here...


--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=C3=A1rabos Coop. V.   Enjoy Data
 "-"

[Numpy-discussion] ISP Changeover complete

[Jeff S. <js...@en...>]

Good evening,

We have completed our internet service switchover. Thank you for your 
patience. If you have any trouble, please let me know.

Thank you,
Jeff Strunk
IT Administrator
Enthought, Inc.

Re: [Numpy-discussion] what is nan_to_num() for float32 ?

[Alan G I. <ai...@am...>]

On Tue, 12 Sep 2006, Sebastian Haase apparently wrote: 
> I have a nice ndarray image viewer built on OpenGL 

Please post (with license).

Thanks!
Alan Isaac

[Numpy-discussion] ISP changeover beginning in 30 minutes (7:00 PM Central)

[Jeff S. <js...@en...>]

Good evening,

We will begin switching our internet service over in about 30 minutes. Please 
allow for up to two hours of downtime. I will send an email announcing the 
completion of this maintenance.

This will effect all Enthought servers as well as the SciPy server which hosts 
many Open Source projects.

We apologize for the inconvenience.
Jeff Strunk
Enthought, Inc.

Re: [Numpy-discussion] what is nan_to_num() for float32 ?

[Sebastian H. <ha...@ms...>]

On Tuesday 12 September 2006 13:33, Sebastian Haase wrote:
> Hi !
> I have a nice ndarray image viewer built on OpenGL - one annoyance though
> is that is crashes if the array contains any NaN, or inf, ...
>
> So, I found N.nan_to_num  - but OpenGL (read: video cards) supports only
> single precision float (N.float32)
>
> So I get this:
> >>> N.nan_to_num([N.inf])
>
> [  1.79769313e+308]
>
> >>> N.nan_to_num([N.inf]).astype(N.float32)
>
> [              inf]
>
> Could nan_to_num() get an optional dtype argument ?
>
> Thanks,
> Sebastian Haase

OK - sorry for the noise...  this is how to do it:

>>> N.nan_to_num(  N.array([N.nan,N.inf,-N.inf],N.float32)  )
[  0.00000000e+00   3.40282347e+38  -3.40282347e+38]


In the meantime I noticed that there is a 
N.asarray_chkfinite() function
but no equivalent for 
N.asanyarray() !

Should   N.asanyarray_chkfinite()  be added  !?

-Sebastian

[Numpy-discussion] what is nan_to_num() for float32 ?

[Sebastian H. <ha...@ms...>]

Hi !
I have a nice ndarray image viewer built on OpenGL - one annoyance though is 
that is crashes if the array contains any NaN, or inf, ...

So, I found N.nan_to_num  - but OpenGL (read: video cards) supports only 
single precision float (N.float32)
So I get this:
>>> N.nan_to_num([N.inf])
[  1.79769313e+308]
>>> N.nan_to_num([N.inf]).astype(N.float32)
[              inf]

Could nan_to_num() get an optional dtype argument ?

Thanks,
Sebastian Haase

Re: [Numpy-discussion] In-place operations

[Travis O. <oli...@ee...>]

Francesc Altet wrote:

>>>>Timer("a=numpy.array(100,dtype=numpy.complex128)", "import
>>>>        
>>>>
>numpy").repeat(3,10000)
>[0.19819307327270508, 0.14915895462036133, 0.14999985694885254]
>  
>
>>>>Timer("a=numpy.array(10000,dtype=numpy.complex128)", "import
>>>>        
>>>>
>numpy").repeat(3,10000)
>[0.15171599388122559, 0.14998698234558105, 0.14901280403137207]
>
>that is 15 us (in my old machine) irregardingly of the size.
>  
>
Ummm..    You are not creating empty arrays here.  You are creating a 
0-d array with a single entry. 

>[BTW, numpy.empty seems twice as slower in my machine. Why?
>  
>
>>>>Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import
>>>>        
>>>>
>numpy").repeat(3,10000)
>[0.37033700942993164, 0.31780219078063965, 0.31607294082641602]
>]
>  
>
Now, you are creating an empty array with 10000 elements in it. 

Best,

-Travis

Re: [Numpy-discussion] In-place operations

[Francesc A. <fa...@ca...>]

El dt 12 de 09 del 2006 a les 13:17 -0400, en/na Pierre Thibault va
escriure:
> Hello again,
>=20
> On 9/12/06, Francesc Altet <fa...@ca...> wrote:
> > Hello Pierre,
> > [...]
> >
> > Well, in some way, there is a temporary array creation that is
> > immediately bound to B, so in the end, the temporary is not so
> > temporary, but a new (bounded) object. Obviously, the object that was
> > referencing B is freed (unless there is another reference to it).
>=20
> ok, I guess I was aware of all that. My worries are related to two cases:
> 1) When the mere creation of a new array is prohibitive.

As Archibald said in other message, creation of a big array is not an
issue (malloc is very fast) and indepent of the size:

>>> Timer("a=3Dnumpy.array(100,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.19819307327270508, 0.14915895462036133, 0.14999985694885254]
>>> Timer("a=3Dnumpy.array(10000,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.15171599388122559, 0.14998698234558105, 0.14901280403137207]

that is 15 us (in my old machine) irregardingly of the size.

[BTW, numpy.empty seems twice as slower in my machine. Why?
>>> Timer("a=3Dnumpy.empty(10000,dtype=3Dnumpy.complex128)", "import
numpy").repeat(3,10000)
[0.37033700942993164, 0.31780219078063965, 0.31607294082641602]
]

> 2) When what I really want to change is the _content_ of an array.
> Using assignment (=3D) disconnects B from the array it refers to, so
> that what used to be true (C=3DB) is not anymore.
>=20
> I understant that B[:] =3D ...  solves the problem 2), though I don't
> know if this notation is recommended; but I would like to know if
> there is anyway to solve 1), in the way ufuncs can do.
>=20
> I had fun using kde's "performance monitor" (ksysguard) to see the
> difference between
>=20
> a =3D numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048)
> a =3D numpy.exp(a)
>=20
> and
>=20
> a =3D numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048)
> numpy.exp(a,out=3Da)
>=20
> Not only is the latter faster, but I could see a large glitch in the
> memory usage for the former.

Yes, it seems that some ufuncs have an additional "out" parameter that I
was not aware of. Well, it that case, this parameter in the fft function
would solve your needs, although I don't know how complicated would be
this.

Cheers,

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=C3=A1rabos Coop. V.   Enjoy Data
 "-"

Re: [Numpy-discussion] In-place operations

[A. M. A. <per...@gm...>]

On 12/09/06, Pierre Thibault <thi...@ph...> wrote:

> I would like to have information on the best techniques to do in-place
> calculations and to minimize temporary array creations. To me this
> seems to be very important whenever the arrays become very large.

The first rule of optimization: don't do it yet.

You can usually go through and banish temporary arrays (using ufuncs
and so on) at the cost of readability, code encapsulation, and
thread-safe-ness. But it may not do what you want. I had an
image-processing code that was taking longer than I thought it should
and using two hundred megabytes or so of RAM. So I rewrote it, with a
certain amount of pain, in a way that it used the fewest possible
temporary arrays. It didn't run any faster, and it then took five
hundred megabytes. Because all the arrays ended up being in memory at
once, the memory footprint increased drastically.

malloc() is fast, typically just a handful of instructions; if you're
allocating a giant array, it's almost certainly being allocated using
mmap(), and it can be released back to the OS on deallocation.

But you probably still want to avoid temporary arrays. So:

> More specifically, here are examples that occured in my code
>
> 1) FFTs:  Let A and B be two large arrays, already allocated. I want
> the fft of A to be stored in B. If I just type B = fft(A), there is a
> temprary array creation, right? Is it possible to avoid that?

Doing an FFT in-place is a major challenge, and involves its own
slowdowns, so generally high-level toolkits don't bother. But fft
seems to be like many functions (those generated by interp1d, for
example) that insist on malloc()ing their own arrays to return. Short
of rooting around in the numpy/scipy code, there's no real way around
this for such functions. The best you can do is make actual use of the
allocated array (rather than copy its contents to *another* array and
discard it).

> 2) Function output: In general, I think the same thing happens with
> functions like
>
> def f1(array_in):
>    array_out = # something using array_in
>    return array_out
>
> Then, if B is already allocated, writing B = f1(A) involves again a
> temporary array creation

Uh, no, not really. The way you have written f1, it probably malloc()s
space for array_out. the address of that space (roughly) is saved in
the array_out variable. If you write B=f1(A), you are just storing the
address in B. The memory is not copied. Even if you do B=f1(A)[::10]
you don't copy the memory.

> I thought instead of doing something like
>
> def f2(array_in, array_out):
>   array_out[:] = # something
>   # Is this good practice?
>
> and call f2(A,B).

This is a ufunc-like solution; you could even make array_out an
optional argument, and return it.

> If I understand well, this still requires a temporary array creation.
> Is there another way of doing that (appart from actually looping
> through the indices of A and B)?

It depends what #something is. If, say, it is 2*array_in, you can
simply do multiply(array_in,2,array_out) to avoid any dynamic
allocation.

> I guess these considerations are not standard python problems because
> you expect python to take care of memory issues. With big arrays in
> scientific computations, I feel the question is more relevant. I might
> be wrong...

Some of these issues come up when dealing with mutable objects (lists,
dictionaries, and so on). Some of them (the fact that python variables
contain simply references) are discussed in various python FAQs.

A. M. Archibald

Re: [Numpy-discussion] In-place operations

[Pierre T. <thi...@ph...>]

Hello again,

On 9/12/06, Francesc Altet <fa...@ca...> wrote:
> Hello Pierre,
> [...]
>
> Well, in some way, there is a temporary array creation that is
> immediately bound to B, so in the end, the temporary is not so
> temporary, but a new (bounded) object. Obviously, the object that was
> referencing B is freed (unless there is another reference to it).

ok, I guess I was aware of all that. My worries are related to two cases:
1) When the mere creation of a new array is prohibitive.
2) When what I really want to change is the _content_ of an array.
Using assignment (=) disconnects B from the array it refers to, so
that what used to be true (C=B) is not anymore.

I understant that B[:] = ...  solves the problem 2), though I don't
know if this notation is recommended; but I would like to know if
there is anyway to solve 1), in the way ufuncs can do.

I had fun using kde's "performance monitor" (ksysguard) to see the
difference between

a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048)
a = numpy.exp(a)

and

a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048)
numpy.exp(a,out=a)

Not only is the latter faster, but I could see a large glitch in the
memory usage for the former.

Pierre

-- 
Pierre Thibault
616 Clark Hall, Cornell University
(607) 255-5522

[Numpy-discussion] numpy.distutils compatibility with subversion-1.4

[Darren D. <dd...@co...>]

This morning I updated to subversion-1.4. There have been several significant 
enhancements in this release (see 
http://subversion.tigris.org/svn_1.4_releasenotes.html), and it appears that 
numpy.distutils is not compatible with working copies created with the new 
svn. I submitted a ticket (#276), but I figured it was worth a post here, 
just a "heads up" for anyone not watching the trac website.

$ python setup.py build
Running from numpy source directory.
non-existing path in 'numpy/distutils': 'site.cfg'
No module named __svn_version__
Traceback (most recent call last):
  File "setup.py", line 89, in ?
    setup_package()
  File "setup.py", line 82, in setup_package
    configuration=configuration )
  File "/home/darren/src/numpy/numpy/distutils/core.py", line 144, in setup
    config = configuration()
  File "setup.py", line 48, in configuration
    config.add_subpackage('numpy')
  File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 753, in 
add_subpackage
    caller_level = 2)
  File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 736, in 
get_subpackage
    caller_level = caller_level + 1)
  File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 683, in 
_get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "./numpy/setup.py", line 8, in configuration
    config.add_subpackage('f2py')
  File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 753, in 
add_subpackage
    caller_level = 2)
  File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 736, in 
get_subpackage
    caller_level = caller_level + 1)
  File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 683, in 
_get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy/f2py/setup.py", line 39, in configuration
    config.make_svn_version_py()
  File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 1298, in 
make_svn_version_py
    self.add_data_files(('', generate_svn_version_py()))
  File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 1281, in 
generate_svn_version_py
    assert revision is not None,'hmm, why I am not inside SVN tree???'
AssertionError: hmm, why I am not inside SVN tree???

Darren

Re: [Numpy-discussion] In-place operations

[Francesc A. <fa...@ca...>]

Hello Pierre,

El dt 12 de 09 del 2006 a les 09:52 -0400, en/na Pierre Thibault va
escriure:
> Hi,
>=20
> I would like to have information on the best techniques to do in-place
> calculations and to minimize temporary array creations. To me this
> seems to be very important whenever the arrays become very large.
>=20
> I already know about the ufunc in-place functionality (which is great).
>=20
> More specifically, here are examples that occured in my code
>=20
> 1) FFTs:  Let A and B be two large arrays, already allocated. I want
> the fft of A to be stored in B. If I just type B =3D fft(A), there is a
> temprary array creation, right? Is it possible to avoid that?

Well, in some way, there is a temporary array creation that is
immediately bound to B, so in the end, the temporary is not so
temporary, but a new (bounded) object. Obviously, the object that was
referencing B is freed (unless there is another reference to it).

>=20
> 2) Function output: In general, I think the same thing happens with
> functions like
>=20
> def f1(array_in):
>    array_out =3D # something using array_in
>    return array_out
>=20
> Then, if B is already allocated, writing B =3D f1(A) involves again a
> temporary array creation

Again, it depends what do you understand by 'temporary'.

>=20
> I thought instead of doing something like
>=20
> def f2(array_in, array_out):
>   array_out[:] =3D # something
>   # Is this good practice?
>=20
> and call f2(A,B).
>=20
> If I understand well, this still requires a temporary array creation.
> Is there another way of doing that (appart from actually looping
> through the indices of A and B)?

Here I'd say that yes, you are creating a truly temporary object and
then assign element by element to B.

>=20
> I know that the use of f2 has one clear advantage: it makes sure that
> whatever was in B is discarded. With f1, the following could happen:
>=20
> A # contains something
> B # contains something
> C =3D B
> B =3D f1(A)
>=20
> C still contains whatever was in B. This could be what you wanted, but
> if you consider C just as a reference to B, this is not good.

This is not good if you want to get rid of the object pointed by B, but
in general, this is considered a nice feature of python.

>=20
> I guess these considerations are not standard python problems because
> you expect python to take care of memory issues. With big arrays in
> scientific computations, I feel the question is more relevant. I might
> be wrong...

If you are worried about wasting memory, just get familiar with this
rule: an object only exists in memory when it is referenced (bounded) by
a variable. When the object is no longer referenced, its memory becomes
freed and is available to the system for later reuse. With this,

B =3D fft(A)

will create a new object (the FFT of A), the object pointed by B will be
freed (if there is not any other reference to it) and the new object
will be bound to B.

If what you want is to avoid having in memory the three objects (namely
A, old B and new B) at the same time, you can do something like:

del B   # deletes reference to object pointed by B
B =3D fft(A)  # B gets bounded to new FFT object

HTH,

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=C3=A1rabos Coop. V.   Enjoy Data
 "-"

[Numpy-discussion] REMINDER: ISP changeover today 9/12 7:00 PM Central

[Jeff S. <js...@en...>]

Good morning,

This is a reminder for this evening's network maintenance.

Unfortunately, our recent change in internet service providers is not working 
out. We will be switching to a more reliable provider tonight at 7:00 
PM Central. Please allow for up to two hours of downtime. I will send an 
email announcing the start and completion of this maintenance.

This will effect all Enthought servers as well as the SciPy server which hosts 
many Open Source projects.

Please pass this message along to people that I have missed. If you have any 
questions, please direct them to me.

We apologize for the inconvenience.
Jeff Strunk
Enthought, Inc.

[Numpy-discussion] In-place operations

[Pierre T. <thi...@ph...>]

Hi,

I would like to have information on the best techniques to do in-place
calculations and to minimize temporary array creations. To me this
seems to be very important whenever the arrays become very large.

I already know about the ufunc in-place functionality (which is great).

More specifically, here are examples that occured in my code

1) FFTs:  Let A and B be two large arrays, already allocated. I want
the fft of A to be stored in B. If I just type B = fft(A), there is a
temprary array creation, right? Is it possible to avoid that?

2) Function output: In general, I think the same thing happens with
functions like

def f1(array_in):
   array_out = # something using array_in
   return array_out

Then, if B is already allocated, writing B = f1(A) involves again a
temporary array creation

I thought instead of doing something like

def f2(array_in, array_out):
  array_out[:] = # something
  # Is this good practice?

and call f2(A,B).

If I understand well, this still requires a temporary array creation.
Is there another way of doing that (appart from actually looping
through the indices of A and B)?

I know that the use of f2 has one clear advantage: it makes sure that
whatever was in B is discarded. With f1, the following could happen:

A # contains something
B # contains something
C = B
B = f1(A)

C still contains whatever was in B. This could be what you wanted, but
if you consider C just as a reference to B, this is not good.

I guess these considerations are not standard python problems because
you expect python to take care of memory issues. With big arrays in
scientific computations, I feel the question is more relevant. I might
be wrong...

Pierre

-- 
Pierre Thibault
616 Clark Hall, Cornell University
(607) 255-5522

[Numpy-discussion] creating a masked array with a sequence containing a masked scalar array

[Michael S. <mic...@gm...>]

It seems that ma.array does not accept a sequence which contains a
masked scalar array (not sure if this is the correct term). Is this
deliberate?

#numpy 1.0b5, python 2.3.x, winXP
import numpy as N
print N.ma.array([1,2,3, N.ma.array(1, mask=True)])

Traceback (most recent call last):
  File "C:\eclipse\plugins\org.python.pydev.debug_1.2.2\pysrc\pydevd_comm.py",
line 529, in doIt
    result = pydevd_vars.evaluateExpression( self.thread_id,
self.frame_id, self.expression, self.doExec )
  File "C:\eclipse\plugins\org.python.pydev.debug_1.2.2\pysrc\pydevd_vars.py",
line 258, in evaluateExpression
    exec expression in frame.f_globals, frame.f_locals
  File "<string>", line 1, in ?
  File "C:\Python23\Lib\site-packages\numpy\core\ma.py", line 562, in __init__
    c = numeric.array(data, dtype=tc, copy=True, order=order)
TypeError: an integer is required

#works fine if scalar masked array is not masked
print N.ma.array([1,2,3, N.ma.array(1)])

[1 2 3 1]

[Numpy-discussion] =?GB2312?B?tPq/qreixrE=?=

[<dg...@to...>]

尊敬的公司领导：(经理/财务)您好！
  
    我司每月有一部分增值税电脑发票和普通商品销售税发票（国税、地税）.优惠代开

或合作，点数较低，（增值税电脑发票6%：普通商品销售发票1-1.5%），还可以根据所做

数量额度的大小来商讨优惠的点数。
   
    本公司郑重承诺所用绝对是真票！更希望能够有机会与贵司合作！验票后付款。诚

信与保密。贵司如有需要欢迎您来电咨询。
　
   联系电话：13590116835
    
   负责人：张豪兴

    E- MAIL ：szh...@12...

    地    址：深圳市深南中路国际文化大厦       

          注：（此信息长期有效敬请保留、如有打扰请原谅。）　

      致
礼！

Re: [Numpy-discussion] ndarray.count() ?

[rex <re...@no...>]

Robert Kern <rob...@gm...> [2006-09-08 06:51]:
> rex wrote:
> > Robert Kern <rob...@gm...> [2006-09-07 16:35]:
> >> rex wrote:
> >>> This exposed inconsistent randint() behavior between SciPy and the Python
> >>> random module. The Python randint includes the upper endpoint. The SciPy
> >>> version excludes it.
> > 
> > I'm not in a position to argue the merits, but IMHO, when code that
> > previously worked silently starts returning subtly bad results after
> > importing numpy, there is a problem. What possible upside is there in
> > having randint() behave one way in the random module and silently behave
> > differently in numpy? 
> 
> I don't understand you. 

That's because I wasn't making any sense. :(

> Importing numpy does not change the standard library's 
> random module in any way. There is no silent difference in behavior. If you use 
> numpy.random you get one set of behavior. If you use random, you get another. 
> Pick the one you want. They're not interchangeable, and nothing suggests that 
> they ought to be.

Of course you're right. I thought the name would be overwritten, and it isn't.
Sorry for wasting your time. :(

Thanks,

-rex

Re: [Numpy-discussion] sum returns grand total?

[Alan G I. <ai...@am...>]

On Mon, 11 Sep 2006, Travis Oliphant apparently wrote: 
> Yes,  you need to specify which axis you want to sum over 
> if you have a multi-dimensional array.  This is the 
> "basic" behavior of most functions 

And a good behavior indeed!

Cheers,
Alan Isaac

Re: [Numpy-discussion] sum returns grand total?

[Travis O. <oli...@ee...>]

lis...@ma... wrote:

>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>I must have missed something -- since when does summing a  
>multidimensional array return the grand total? I thought that it  
>should sum over axis zero by default (at least, thats what the docs  
>say). Here's what I am getting:
>
>(Pdb) pre
>Out[6]:
>array([[[ 13,   3,   1,  33,  52],
>         [  1,   0,   0,   0,   2],
>         [  4,   0,   0,   2,   6],
>         [ 15,   1,   0,  20,  54]],
>
>        [[380, 101,   9, 267, 321],
>         [ 20,   4,   0,   7,  16],
>         [ 31,   3,   1,  31,  22],
>         [314,  27,   4, 276, 391]]], dtype=int64)
>(Pdb) sum(pre)
>Out[6]: 2432L
>
>Is this correct behaviour?
>  
>

Yes,  you need to specify which axis you want to sum over if you have a 
multi-dimensional array.  This is the "basic" behavior of most functions 
in numpy unless you import them from oldnumeric.   See the 
fix_default_axis.py file for help on converting code that worked for 
numpy 0.9.8 to 1.0 where this switch was made. 

-Travis

[Numpy-discussion] sum returns grand total?

[<lis...@ma...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I must have missed something -- since when does summing a  
multidimensional array return the grand total? I thought that it  
should sum over axis zero by default (at least, thats what the docs  
say). Here's what I am getting:

(Pdb) pre
Out[6]:
array([[[ 13,   3,   1,  33,  52],
         [  1,   0,   0,   0,   2],
         [  4,   0,   0,   2,   6],
         [ 15,   1,   0,  20,  54]],

        [[380, 101,   9, 267, 321],
         [ 20,   4,   0,   7,  16],
         [ 31,   3,   1,  31,  22],
         [314,  27,   4, 276, 391]]], dtype=int64)
(Pdb) sum(pre)
Out[6]: 2432L

Is this correct behaviour?
- --
Christopher Fonnesbeck
+ Atlanta, GA
+ fonnesbeck at mac.com
+ Contact me on AOL IM using email address


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFFBYSJGKaDpEGshJ0RAiPhAJ0d/MCGMciLgB2ZOwXf1FVj26NpTgCdG7h8
M83+mZsYGP2svVBGn8RtbNw=
=mby0
-----END PGP SIGNATURE-----

Re: [Numpy-discussion] rfft different in numpy vs scipy

[Andrew J. <a.h...@ba...>]

Steven G. Johnson wrote:
> Andrew Jaffe wrote:
>> numpy returns n/2+1 complex numbers (so the first and last numbers are
>> actually real) with the frequencies equivalent to the positive part of
>> the fftfreq, whereas scipy returns n real numbers with the frequencies
>> as in rfftfreq (i.e., two real numbers at the same frequency, except for
>> the highest and lowest) [All of the above for even n; but the difference
>> between numpy and scipy remains for odd n.]
>>
>> I think the numpy behavior makes more sense, as it doesn't require any
>> unpacking after the fact, at the expense of a tiny amount of wasted
>> space. But would this in fact require scipy doing extra work from
>> whatever the 'native' real_fft (fftw, I assume) produces?
> 
> As an author of FFTW, let me interject a couple of points into this
> discussion.
> 
> First, if you are using FFTW, then its real-input r2c routines
> "natively" produce output in the "unpacked" numpy format as described
> above: an array of n/2+1 complex numbers.  Any "packed" format would
> require some data permutations.  Other FFT implementations use a
> variety of formats.
> 
> Second, the *reason* why FFTW's r2c routines produce unpacked output is
> largely because "packed" formats do not generalize well to
> multi-dimensional FFTs, while the "unpacked" format does.  (Packed
> formats are *possible* for multidimensional transforms, but become
> increasingly intricate as you add more dimensions.)  Additionally, I
> personally find the unpacked format more convenient in most
> applications.
> 
> I hope this is helpful.

OK -- so it appears that all (three) of the votes so far are in support 
of the numpy convention -- a complex result.

If this is something that can be done in pure python, I'm willing to 
give it a stab, but I'm probably not capable of handling any python/C 
issues. Does anyone out there understand the interaction between fftpack 
(C & python?), fftw (C), scipy and numpy in this context well enough to 
give some advice?

Yours,

Andrew

Re: [Numpy-discussion] rfft different in numpy vs scipy

[Steven G. J. <st...@al...>]

Andrew Jaffe wrote:
> numpy returns n/2+1 complex numbers (so the first and last numbers are
> actually real) with the frequencies equivalent to the positive part of
> the fftfreq, whereas scipy returns n real numbers with the frequencies
> as in rfftfreq (i.e., two real numbers at the same frequency, except for
> the highest and lowest) [All of the above for even n; but the difference
> between numpy and scipy remains for odd n.]
>
> I think the numpy behavior makes more sense, as it doesn't require any
> unpacking after the fact, at the expense of a tiny amount of wasted
> space. But would this in fact require scipy doing extra work from
> whatever the 'native' real_fft (fftw, I assume) produces?

As an author of FFTW, let me interject a couple of points into this
discussion.

First, if you are using FFTW, then its real-input r2c routines
"natively" produce output in the "unpacked" numpy format as described
above: an array of n/2+1 complex numbers.  Any "packed" format would
require some data permutations.  Other FFT implementations use a
variety of formats.

Second, the *reason* why FFTW's r2c routines produce unpacked output is
largely because "packed" formats do not generalize well to
multi-dimensional FFTs, while the "unpacked" format does.  (Packed
formats are *possible* for multidimensional transforms, but become
increasingly intricate as you add more dimensions.)  Additionally, I
personally find the unpacked format more convenient in most
applications.

(FFTW does actually support a different, "packed" format in its r2r
interface, but only for 1d FFTs.  The reason for this has do to with
advantages of that format for odd sizes.  We recommend that most users
employ the r2c interface, however; our r2c interface is generally
faster for even sizes.)

I hope this is helpful.

Cordially,
Steven G. Johnson

Re: [Numpy-discussion] 1.0b5 build failure

[Fernando P. <fpe...@gm...>]

On 9/9/06, Neal Becker <ndb...@gm...> wrote:
> Sorry, actually that's today's numpy svn - not 1.0b5.

Make sure you have a clean build environment, the current SVN builds
just fine on my Dapper 6.06 box; I just rebuilt it a second ago.

Cheers,

f

ps. Here's my little build script, which I use to rebuild
numpy/scipy/matplotlib in one shot:

longs[scipy]> cat makeall
#!/bin/sh

./makepkg numpy
./makepkg scipy
./makepkg matplotlib

# EOF

which uses the simple makepkg (just edit the PYTHON and PYPREFIX vars
to suit your config):

longs[scipy]> cat makepkg
#!/bin/sh

PACKAGE=$1

PYTHON=python2.4
PYPREFIX=$HOME/tmp/local


#REVISION="HEAD"
#REVISION=2772 # for numpy
#REVISION=2054 # for scipy
#REVISION="{2006-07-06}"
#svn update -r $REVISION ${PACKAGE}

svn update ${PACKAGE}

export PYTHONPATH=${PYPREFIX}/lib/${PYTHON}/site-packages:${PYTHONPATH}:

# remove existing ${PACKAGE} to make sure the build doesn't pick up
spurious things
rm -rf $PYPREFIX/lib/${PYTHON}/site-packages/${PACKAGE}

# make/install
cd ${PACKAGE}
rm -rf build
$PYTHON setup.py install --prefix=$PYPREFIX

Re: [Numpy-discussion] 1.0b5 build failure

[Neal B. <ndb...@gm...>]

Sorry, actually that's today's numpy svn - not 1.0b5.

[Numpy-discussion] 1.0b5 build failure

[Neal B. <ndb...@gm...>]

=46edora FC5 linux, with BLAS, LAPACK, ATLAS, FFTW
compile=20
options: '-Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include =
=2DIbuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/inc=
lude -I/usr/include/python2.4 -c'
gcc: numpy/core/src/multiarraymodule.c
numpy/core/src/multiarraymodule.c: In function =E2=80=98initmultiarray=E2=
=80=99:
numpy/core/src/multiarraymodule.c:6860: error: =E2=80=98NPY_ALLOW_THREADS=
=E2=80=99 undeclared=20
(first use in this function)
numpy/core/src/multiarraymodule.c:6860: error: (Each undeclared identifier =
is=20
reported only once
numpy/core/src/multiarraymodule.c:6860: error: for each function it appears=
=20
in.)
numpy/core/src/multiarraymodule.c: In function =E2=80=98initmultiarray=E2=
=80=99:
numpy/core/src/multiarraymodule.c:6860: error: =E2=80=98NPY_ALLOW_THREADS=
=E2=80=99 undeclared=20
(first use in this function)
numpy/core/src/multiarraymodule.c:6860: error: (Each undeclared identifier =
is=20
reported only once
numpy/core/src/multiarraymodule.c:6860: error: for each function it appears=
=20
in.)
error:=20
Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,=
=2DD_FORTIFY_SOURCE=3D2 -fexceptions -fstack-protector --param=3Dssp-buffer=
=2Dsize=3D4 -m64 -mtune=3Dgeneric -D_GNU_SOURCE -fPIC -fPIC -Ibuild/src.lin=
ux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-=
2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2=
=2E4 -c=20
numpy/core/src/multiarraymodule.c -o=20
build/temp.linux-x86_64-2.4/numpy/core/src/multiarraymodule.o" failed with=
=20
exit status 1