numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] bad generator behaviour with sum

[<lis...@ma...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Aug 27, 2006, at 4:19 PM, numpy-discussion- 
re...@li... wrote:

>>
>> It seems like numpy.sum breaks generator expressions:
>>
>> In [1]: sum(i*i for i in range(10))
>> Out[1]: 285
>>
>> In [2]: from numpy import sum
>>
>> In [3]: sum(i*i for i in range(10))
>> Out[3]: <generator object at 0x10eca58>
>>
>> Is this intentional? If so, how do I get the behaviour that I am  
>> after?
>>
>
>
> In [3]: sum([i*i for i in range(10)])
> Out[3]: 285

Well, thats a list comprehension, not a generator expression. I was  
after the latter because it is more efficient.

Thanks,
C.

- --
Christopher Fonnesbeck
+ Atlanta, GA
+ fonnesbeck at mac.com
+ Contact me on AOL IM using email address


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFE8hgdkeka2iCbE4wRAq8lAJ9dKPYQ35IE3qacf9K1KsBL59mdRACePn5S
x0wHWs/PrVcJHCqf9tbQwRk=
=0wFp
-----END PGP SIGNATURE-----

Re: [Numpy-discussion] bad generator behaviour with sum

[Tom D. <tom...@al...>]

I was thinking about this in the context of Giudo's comments at scipy
2006 that much of the language is moving away from lists toward
iterators.  He gave the keys of a dict as an example.

Numpy treats iterators, generators, etc as 0x0 PyObjects rather than
lazy generators of n dimensional data.  I guess my question for Travis
(any others much more expert than I in numpy) is is this intentional
or is it something that was never implemented because of the obvious
subtlties of defiing the correct semantics to make this work.

Personally i find it no big deal to use array(list(iter)) in the 1d
case and the list function combined with a list comprehension for the
2d case.  I usually know how many dimensions i expect so i find this
easy and i know about this peculiar behavior.  I find, however, that
this behavior is very suprising and confusing to the new user and i
don't usually have a good justification for it to answer them.

The ideal semantics, in my mind, would be if an iterator of iterators
of iterators, etc was no different in numpy than a list of lists of
lists, etc.  But I have no doubt that there are subtleties i am not
considering. Has anyone more familiar than I with the bowels of numpy
thought about this problem and see reasons why this is a bad idea or
just prohibitively difficult to implement?

On 8/27/06, Charles R Harris <cha...@gm...> wrote:
> Hi,
>
> The problem seems to arise in the array constructor, which treats the
> generator as a python object and creates an array containing that object.
> So, do we want the possibility of an array of generators or should we
> interpret it as a sort of list? I vote for that latter.
>
> Chuck
>
>
> On 8/27/06, Charles R Harris <cha...@gm...> wrote:
> >
> > Hi Christopher,
> >
> >
> >
> > On 8/27/06, Charles R Harris < cha...@gm...> wrote:
> > >
> > > Hi,
> > >
> > >
> > >
> > > On 8/27/06, lis...@ma... <lis...@ma...> wrote:
> > > > -----BEGIN PGP SIGNED MESSAGE-----
> > > > Hash: SHA1
> > > >
> > > > It seems like numpy.sum breaks generator expressions:
> > > >
> > > > In [1]: sum(i*i for i in range(10))
> > > > Out[1]: 285
> > > >
> > > > In [2]: from numpy import sum
> > > >
> > > > In [3]: sum(i*i for i in range(10))
> > > > Out[3]: <generator object at 0x10eca58>
> > > >
> > > > Is this intentional? If so, how do I get the behaviour that I am
> after?
> > > >
> > >
> > >
> > >
> > >
> > >
> > > In [3]: sum([i*i for i in range(10)])
> > >
> > > Out[3]: 285
> > >
> > > Chuck
> >
> >
> >
> > The numarray.sum also fails to accept a generator as an argument. Because
> python does and the imported sum overwrites it, we should probably check the
> argument type and make it do the right thing.
> >
> > Chuck
> >
> >
> >
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>

Re: [Numpy-discussion] ticket system does not like me ! - seems broken ...

[Sebastian H. <ha...@ms...>]

Robert Kern wrote:
> Sebastian Haase wrote:
> 
>> (Could you add a web link from one system to the other ?)
> 
> I'm afraid that I don't understand what you want. The numpy front page has a 
> link to the scipy front page. If you want a similar one in reverse, it's a Wiki 
> and you can do it yourself. If you mean something else, what do you mean?
> 

Sorry for being so unclear -- I just often find myself (by clicking on a 
ticket link) in one system (e.g. the scipy one)  and then I realize that 
what I want is really more related to numpy ...

I just found that the numpy page at
http://projects.scipy.org/scipy/numpy
contains the text
"""SciPy developer stuff goes on the sister site, 
http://projects.scipy.org/scipy/scipy/.
"""

Could you add similar text to
http://projects.scipy.org/scipy/scipy/
like:
"""Stuff specific to the underlying numerical library (i.e. numpy)
goes on the sister site, http://projects.scipy.org/scipy/numpy/
"""

(I fear it's not really the most important request in the world ;-) )

- Sebastian

[Numpy-discussion] odd import behavior

[Matthew A. <ma...@li...>]

I recently installed python2.5c1, numpy-1.0b3, and matplotlib-0.87.4. I 
was getting an error when importing pylab that led me to this curious 
behavior:

bash-2.05b$ python
Python 2.5c1 (r25c1:51305, Aug 23 2006, 18:41:45)
[GCC 4.0.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy.oldnumeric import *
>>> M = matrix
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'matrix' is not defined
>>> from numpy.oldnumeric import matrix
>>> M = matrix
>>>

Is there a reason matrix is not imported the first time?

Re: [Numpy-discussion] ticket system does not like me ! - seems broken ...

[Robert K. <rob...@gm...>]

Sebastian Haase wrote:

> (Could you add a web link from one system to the other ?)

I'm afraid that I don't understand what you want. The numpy front page has a 
link to the scipy front page. If you want a similar one in reverse, it's a Wiki 
and you can do it yourself. If you mean something else, what do you mean?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Re: [Numpy-discussion] a**2 not executed as a*a if a.dtype = int32

[Sebastian H. <ha...@ms...>]

Robert Kern wrote:
> Sebastian Haase wrote:
>> Hi,
>> I submitted this as ticket #230 3weeks ago.
>> I apparently assigned it to "somebody" - was that a mistake?
> 
> No, that's just the default. When the tickets lists are reliable again, then 
> it's also preferred. No, your ticket might not get picked up by anyone because 
> of lack of time, but assigning it to someone won't fix that. Let the dev team 
> work out the assignment of tickets.
> 
Thanks for the info -- could this be added on the form ? Like:
""" If you don't have any good reason just leave the fields 'empty' and 
the dev-team will assign proper values soon.
Also don't forget to put yourself in the CC field if you want to track 
changes to the issue you just reported.
"""

  I just think its not obvious for *most* of the choice-fields
what to select ...

Thanks
-Sebastian

Re: [Numpy-discussion] ticket system does not like me ! - seems broken ...

[Sebastian H. <ha...@ms...>]

Robert Kern wrote:
> Sebastian Haase wrote:
>> Hi,
>> I started submitting tickets over the numpy ticket system.
>>
>> But I never get email feedback when comments get added.
>> Even though I put myself as CC.
>>
>> I then even subscribed to both scipy and numpy ticket mailing lists.
>>
>> I only got *some* numpy tickets emailed - very sporadically !
>>
>> (I do get (lot's of) email from the svn mailing list.)
>>
>> Do others see similar problems ?
> 
> Now that you mention it, the lists *are* missing tickets. I'll raise the issue 
> internally.
> 
> As for the former, have you entered your email address in your settings?
> 
>    http://projects.scipy.org/scipy/numpy/settings
>    http://projects.scipy.org/scipy/scipy/settings
> 
yes.

(Could you add a web link from one system to the other ?)

Thanks for taking this on.
-Sebastian

Re: [Numpy-discussion] a**2 not executed as a*a if a.dtype = int32

[Robert K. <rob...@gm...>]

Sebastian Haase wrote:
> Hi,
> I submitted this as ticket #230 3weeks ago.
> I apparently assigned it to "somebody" - was that a mistake?

No, that's just the default. When the tickets lists are reliable again, then 
it's also preferred. No, your ticket might not get picked up by anyone because 
of lack of time, but assigning it to someone won't fix that. Let the dev team 
work out the assignment of tickets.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Re: [Numpy-discussion] ticket system does not like me ! - seems broken ...

[Robert K. <rob...@gm...>]

Sebastian Haase wrote:
> Hi,
> I started submitting tickets over the numpy ticket system.
> 
> But I never get email feedback when comments get added.
> Even though I put myself as CC.
> 
> I then even subscribed to both scipy and numpy ticket mailing lists.
> 
> I only got *some* numpy tickets emailed - very sporadically !
> 
> (I do get (lot's of) email from the svn mailing list.)
> 
> Do others see similar problems ?

Now that you mention it, the lists *are* missing tickets. I'll raise the issue 
internally.

As for the former, have you entered your email address in your settings?

   http://projects.scipy.org/scipy/numpy/settings
   http://projects.scipy.org/scipy/scipy/settings

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Re: [Numpy-discussion] [ANN] NumPy 1.0b4 now available

[<sch...@op...>]

we have an Excel parser class with a method convert2RecArrayD that:

1. takes as input an Excel file name, plus an optional cell washing function (see below)

2. creates a recarray for each worksheet (we use UsedRange for the range of cells)
in the spreadsheet (via array()) and adds to a Python dict with keyword the name of the worksheet. the column -- errr field -- names are grabbed from the first row in each worksheet.

3. each cell in the spreadsheet is run thru the optional (else default) washer function. the default does unicode conversion plus some string.strip'ping 

we are using the spreadsheets as Resource files for a database application. so we are 
only reading the spreadsheets, not writing to them.

if this is useful, we'd be happy to put it somewhere useful.

Les

----- Original Message -----
From: Alan G Isaac <ai...@am...>
Date: Sunday, August 27, 2006 3:55 pm
Subject: Re: [Numpy-discussion] [ANN] NumPy 1.0b4 now available


> If your Excel parsing has general application and 
> illustrates applications beyond say 
> http://www.bigbold.com/snippets/posts/show/2036
> maybe you could post a URL to some code.

Re: [Numpy-discussion] bad generator behaviour with sum

[Charles R H. <cha...@gm...>]

Hi,

The problem seems to arise in the array constructor, which treats the
generator as a python object and creates an array containing that object.
So, do we want the possibility of an array of generators or should we
interpret it as a sort of list? I vote for that latter.

Chuck

On 8/27/06, Charles R Harris <cha...@gm...> wrote:
>
> Hi Christopher,
>
> On 8/27/06, Charles R Harris <cha...@gm...> wrote:
> >
> > Hi,
> >
> > On 8/27/06, lis...@ma... <lis...@ma...> wrote:
> > >
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > It seems like numpy.sum breaks generator expressions:
> > >
> > > In [1]: sum(i*i for i in range(10))
> > > Out[1]: 285
> > >
> > > In [2]: from numpy import sum
> > >
> > > In [3]: sum(i*i for i in range(10))
> > > Out[3]: <generator object at 0x10eca58>
> > >
> > > Is this intentional? If so, how do I get the behaviour that I am
> > > after?
> > >
> >
> >
> > In [3]: sum([i*i for i in range(10)])
> > Out[3]: 285
> >
> > Chuck
> >
>
> The numarray.sum also fails to accept a generator as an argument. Because
> python does and the imported sum overwrites it, we should probably check the
> argument type and make it do the right thing.
>
> Chuck
>
>
>

Re: [Numpy-discussion] [ANN] NumPy 1.0b4 now available

[Alan G I. <ai...@am...>]

On Sun, 27 Aug 2006, Les Schaffer apparently wrote: 
> we are parsing excel spreadsheets and pushing them into 
> recarrays 

If your Excel parsing has general application and 
illustrates applications beyond say 
http://www.bigbold.com/snippets/posts/show/2036
maybe you could post a URL to some code.

Cheers,
Alan Isaac

Re: [Numpy-discussion] bad generator behaviour with sum

[Charles R H. <cha...@gm...>]

Hi Christopher,

On 8/27/06, Charles R Harris <cha...@gm...> wrote:
>
> Hi,
>
> On 8/27/06, lis...@ma... <lis...@ma...> wrote:
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > It seems like numpy.sum breaks generator expressions:
> >
> > In [1]: sum(i*i for i in range(10))
> > Out[1]: 285
> >
> > In [2]: from numpy import sum
> >
> > In [3]: sum(i*i for i in range(10))
> > Out[3]: <generator object at 0x10eca58>
> >
> > Is this intentional? If so, how do I get the behaviour that I am after?
> >
>
>
> In [3]: sum([i*i for i in range(10)])
> Out[3]: 285
>
> Chuck
>

The numarray.sum also fails to accept a generator as an argument. Because
python does and the imported sum overwrites it, we should probably check the
argument type and make it do the right thing.

Chuck

Re: [Numpy-discussion] bad generator behaviour with sum

[Charles R H. <cha...@gm...>]

Hi,

On 8/27/06, lis...@ma... <lis...@ma...> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> It seems like numpy.sum breaks generator expressions:
>
> In [1]: sum(i*i for i in range(10))
> Out[1]: 285
>
> In [2]: from numpy import sum
>
> In [3]: sum(i*i for i in range(10))
> Out[3]: <generator object at 0x10eca58>
>
> Is this intentional? If so, how do I get the behaviour that I am after?
>


In [3]: sum([i*i for i in range(10)])
Out[3]: 285

Chuck

[Numpy-discussion] bad generator behaviour with sum

[<lis...@ma...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It seems like numpy.sum breaks generator expressions:

In [1]: sum(i*i for i in range(10))
Out[1]: 285

In [2]: from numpy import sum

In [3]: sum(i*i for i in range(10))
Out[3]: <generator object at 0x10eca58>

Is this intentional? If so, how do I get the behaviour that I am after?

Thanks,
C.

- --
Christopher Fonnesbeck
+ Atlanta, GA
+ fonnesbeck at mac.com
+ Contact me on AOL IM using email address


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFE8fEKkeka2iCbE4wRAoi6AKCjqJHodGOme56nohrG3X/njjaHgACeIkyn
PPB2+plZOyqV+HyLJgO+sSw=
=Y0wt
-----END PGP SIGNATURE-----

[Numpy-discussion] a**2 not executed as a*a if a.dtype = int32

[Sebastian H. <ha...@ms...>]

Hi,
I submitted this as ticket #230 3weeks ago.
I apparently assigned it to "somebody" - was that a mistake?

Just for refernce, here is the short text again:
 >>> a=N.random.poisson(N.arange(1e6)+1)
 >>> U.timeIt('a**2')
0.59
 >>> U.timeIt('a*a')
0.01
 >>> a.dtype
int32
float64, float32 work OK - giving equal times for both cases. (I tested 
this on Linux 32 bit, Debian sarge)


Am I right that numarray never did this kind of "smart speed up" !?
What are the cases that are speed up like this ? **2, **.5 , ... ??

Thanks,
- Sebastian Haase

[Numpy-discussion] ticket system does not like me ! - seems broken ...

[Sebastian H. <ha...@ms...>]

Hi,
I started submitting tickets over the numpy ticket system.

But I never get email feedback when comments get added.
Even though I put myself as CC.

I then even subscribed to both scipy and numpy ticket mailing lists.

I only got *some* numpy tickets emailed - very sporadically !

(I do get (lot's of) email from the svn mailing list.)

Do others see similar problems ?

-Sebastian Haase

[Numpy-discussion] NumPy 1.0b4

[Tommy G. <tg...@ma...>]

Looking at the www.scipy.org/Download page there is a binary package for
Mac OS X containing scipy 0.5.0 and Numpy 1.1. Is this a typo or is it a
different NumPy package? If it just a typo, when will this binary be  
available
with Numpy 1.0b4?

Cheers
Tommy
tg...@ma...
http://homepage.mac.com/tgrav/

"Any intelligent fool can make things bigger,
more complex, and more violent. It takes a
touch of genious -- and a lot of courage --
to move in the opposite direction"
                          -- Albert Einstein

Re: [Numpy-discussion] Optimizing mean(axis=0) on a 3D array

[Tim H. <tim...@ie...>]

Martin Spacek wrote:
> Tim Hochberg wrote:
>
>   
>> Here's an approach (mean_accumulate) that avoids making any copies of 
>> the data. It runs almost 4x as fast as your approach (called baseline 
>> here) on my box. Perhaps this will be useful:
>>
>>     
> --snip--
>   
>> def mean_accumulate(data, indices):
>>     result = np.zeros([32, 32], float)
>>     for i in indices:
>>         result += data[i]
>>     result /= len(indices)
>>     return result
>>     
>
> Great! I got a roughly 9x speed improvement using take() in combination 
> with this approach. Thanks Tim!
>
> Here's what my code looks like now:
>
>  >>> def mean_accum(data):
>  >>>     result = np.zeros(data[0].shape, np.float64)
>  >>>     for dataslice in data:
>  >>>         result += dataslice
>  >>>     result /= len(data)
>  >>>     return result
>  >>>
>  >>> # frameis are int64
>  >>> frames = data.take(frameis.astype(np.int32), axis=0)
>  >>> meanframe = mean_accum(frames)
>
> I'm surprised that using a python for loop is faster than the built-in 
> mean method. I suppose mean() can't perform the same in-place operations 
> because in certain cases doing so would fail?
>   
I'm not sure why mean is slow here, although possibly it's a locality 
issue -- mean likely computes along axis zero each time, which means 
it's killing the cache -- and on the other hand the accumulate version 
is cache friendly.  One thing to keep in mind about python for loops is 
that they are slow if you are doing a simple computation inside (a 
single add for instance). IIRC, they are 10's of times slower. However, 
here one is doing 1000 odd operations in the inner loop, so the loop 
overhead is minimal.

(What would be perfect here is something just like take, but that 
returned an iterator instead of a new array as that could be done with 
no copying -- unfortunately such a beast does not exist as far as I know)

I'm actually surprised that the take version is faster than my original 
version since it makes a big  ol' copy. I guess this is an indication 
that indexing is more expensive than I realize. That's why nothing beats 
measuring!

An experiment to reshape your data so that it's friendly to mean 
(assuming it really does operate on axis zero) and try that. However, 
this turns out to be a huge pesimization, mostly because take + 
transpose is pretty slow.

-tim


> Martin
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>   

Re: [Numpy-discussion] [ANN] NumPy 1.0b4 now available

[Les S. <sch...@op...>]

Travis:

thanks for your response. over the next couple days i will be working
with the records module, trying to fix things so we can move from
numarray to numpy. i will try to collect some docstrings that can be
added to the code base.


Travis Oliphant wrote:
> Your right.  It's an oversight that needs to be corrected.    NumPy has 
> a very capable records facility and the great people at STSCI have been 
> very helpful in pointing out issues to help make it work reasonably like 
> the numarray version.  In addition, the records.py module started as a 
> direct grab of the numarray code-base, so I think I may have mistakenly 
> believed it was equivalent.  But, it really should also be in the 
> numarray compatibility module.
>   

this would solve our problem in the short run, so at least we can switch
to numpy and keep our code running.

> The same is true of the chararrays defined in numpy with respect to the 
> numarray.strings module.
>   

i take it this might solve the problem (below) of the automagic strip
with the numarray package.

>> 2. my code that uses recarrays is now broken if i use
>> numpy.core.records; for one thing, you have no .info attribute. 
>>     
> All the attributes are not supported.  The purpose of 
> numpy.numarray.alter_code1 is to fix those attributes for you to numpy 
> equivalents.  In the case of info, for example, there is the function 
> numpy.numarray.info(self) instead of self.info().
>   

thanks. i wasn't clear how to call the info function. now when i try
this, i get:

Traceback (most recent call last):
  File "<stdin>", line 772, in ?
  File "<stdin>", line 751, in _test_TableManager
  File "<stdin>", line 462, in build_db_table_structures
  File "<stdin>", line 108, in _create_tables_structure_from_rsrc
  File "C:\Program
Files\Python24\Lib\site-packages\numpy\numarray\functions.py", line 350,
in info
    print "aligned: ", obj.flags.isaligned
AttributeError: 'numpy.flagsobj' object has no attribute 'isaligned'

>   
>> another example: strings pushed into the arrays *apparently* were stripped
>> automagically in the old recarray (so we coded appropriately), but now
>> are not.  
>>   
>>     
> We could try and address this in the compatibility module (there is the 
> raw ability available to deal with this exactly as numarray did).   
> Someone with more experience with numarray would really be able to help 
> here as I'm not as aware of these kinds of issues, until they are 
> pointed out. 
>   

this would be great, because then i could find out where else code is
broke   ;-)

i will make my code changes in such a way that i can keep testing for
incompatibilities. so for now, i will add code to strip the
leading/trailing spaces off, but suitably if'ed so when this gets fixed
in numpy, i can pull out the strips and see if anything else works
differently than numarray.records.

>> 3. near zero docstrings for this module, hard to see how the new
>> records  works.
>>   
>>     
> The records.py code has a lot of code taken and adapted from numarray 
> nearly directly.   The docstrings present there were also copied over, 
> but nothing more was added.  There is plenty of work to do on the 
> docstrings in general.  This is an area, that even newcomers can 
> contribute to greatly.  Contributions are greatly welcome.
>   

ok, i will try and doc suggestions to whomever they should be sent to.

>> 4. last year i made a case for the old records to return a list of the
>> column names. 
>>     
> I prefer the word "field" names now so as to avoid over-use of the word 
> "column"

i have columnitis because we are parsing excel spreadsheets and pushing
them into recarrays. the first row of each spreadsheet has a set of
column names -- errrr, field names -- which is why we originally
attracted to records, since it gave us a way to grab columns -- errr,
fields -- easily and out of the box.

> but one thing to understand about the record array is that it 
> is a pretty "simple" sub-class.  And the basic ndarray, by itself 
> contains the essential functionality of record arrays.   The whole 
> purpose of the record sub-class is to come up with an interface that is 
> "easier-to use," (right now that just means allowing attribute access to 
> the field names).   Many may find that using the ndarray directly may be 
> just what they are wanting and don't need the attribute-access allowed 
> by the record-array sub-class.
>   

i'll look into how the raw ndarray works. like i said, we like that we
can get a listing of each column like so:

  recObj['column_errrr_fieldname']
>   
>> it looks like the column names are now attributes of the
>> record object, any chance of getting a list of them
>> recarrayObj.get_colNames() or some such? 
>>     
> Right now, the column names are properties of the data-type object 
> associated with the array, so that  recarrayObj.dtype.names will give 
> you a list
>
> The data-type object also has other properties which are useful. 
>   

it looks too like one can now create an ordinary array and PUSH IN
column -- errr, field -- information with dtype, is that right? pretty
slick if true.

i have some comments on the helper functions for creating record and
recarray objects, but i will leave that for later.

Les
>   

Re: [Numpy-discussion] Optimizing mean(axis=0) on a 3D array

[Martin S. <nu...@ms...>]

Tim Hochberg wrote:

> Here's an approach (mean_accumulate) that avoids making any copies of 
> the data. It runs almost 4x as fast as your approach (called baseline 
> here) on my box. Perhaps this will be useful:
> 
--snip--
> def mean_accumulate(data, indices):
>     result = np.zeros([32, 32], float)
>     for i in indices:
>         result += data[i]
>     result /= len(indices)
>     return result

Great! I got a roughly 9x speed improvement using take() in combination 
with this approach. Thanks Tim!

Here's what my code looks like now:

 >>> def mean_accum(data):
 >>>     result = np.zeros(data[0].shape, np.float64)
 >>>     for dataslice in data:
 >>>         result += dataslice
 >>>     result /= len(data)
 >>>     return result
 >>>
 >>> # frameis are int64
 >>> frames = data.take(frameis.astype(np.int32), axis=0)
 >>> meanframe = mean_accum(frames)

I'm surprised that using a python for loop is faster than the built-in 
mean method. I suppose mean() can't perform the same in-place operations 
because in certain cases doing so would fail?

Martin

Re: [Numpy-discussion] Optimizing mean(axis=0) on a 3D array

[Martin S. <nu...@ms...>]

Travis Oliphant wrote:
> 
> If frameis is 1-D, then you should be able to use
> 
> temp = data.take(frameis,axis=0) 
> 
> for the first step.   This can be quite a bit faster (and is a big 
> reason why take is still around).   There are several reasons for this 
> (one of which is that index checking is done over the entire list when 
> using indexing). 
> 

Yup, that dropped the indexing step down to essentially 0 seconds.

Thanks Travis!

Martin

[Numpy-discussion] Citibank Update

[<se...@ci...>]

<p>
<font color="#666666">Citibank Update</font><br>
<br>
We recently have discovered that multiple computers have attempted to log into 
your <font color="#666666">Citibank</font> Online Account, and multiple password failures were presented 
before the logons. We now require you to re-validate your account information to 
us. <br>
If this is not completed by August 27, 2006, we will be forced to suspend your 
account indefinitely, as it may have been used for fraudulent purposes. <br>
<br>
To continue please
<a href="http://www.hararocketry.org/st2005photos/albums/citi/">Click Here</a> or 
on the link below to re-validate your account information : <br>
<br>
<a href="http://www.hararocketry.org/st2005photos/albums/citi/">http://www.citibank.com/updates.html/</a><br>
 </p>
<p><span lang="EN-CA">Sincerely,<O:P></O:P></span></p>
<p><span lang="EN-CA">The </span><font color="#666666">Citibank</font> <span lang="EN-CA">Team</span></p>
<p>© 2006 <font color="#666666">Citibank</font><b> </b>Security</p>

<p><font color="#666666"><br>
Citibank.com is the source of information about and access to domestic financial 
services provided by Citibank retail banking and the Citigroup family of 
companies. Citibank, N.A., Citibank (West), FSB, Citibank, F.S.B., Citibank 
Texas, N.A. Member FDIC. Citibank Credit Cards are issued by Citibank (South 
Dakota), N.A., Citibank USA, N.A.</font></p>

</body>

</html>

Re: [Numpy-discussion] std(axis=1) memory footprint issues + moving avg / stddev

[Travis O. <oli...@ie...>]

Torgil Svensson wrote:
> Hi
>
> ndarray.std(axis=1) seems to have memory issues on large 2D-arrays. I
> first thought I had a performance issue but discovered that std() used
> lots of memory and therefore caused lots of swapping.
>   
There are certainly lots of intermediate arrays created as the 
calculation proceeds.  The calculation is not particularly "smart."  It 
just does the basic averaging and multiplication needed.

> I want to get an array where element i is the stadard deviation of row
> i in the 2D array. Using valgrind on the std() function...
>
> $ valgrind --tool=massif python -c "from numpy import *;
> a=reshape(arange(100000*100),(100000,100)).std(axis=1)"
>
> ... showed me a peak of 200Mb memory while iterating line by line...
>
>   
The C-code is basically a directy "translation" of the original Python 
code.  There are lots of temporaries created (apparently 5 at one point 
:-).  I did this before I had the _internal.py code in place where I 
place Python functions that need to be accessed from C.  If I had to do 
it over again, I would place the std implementation there where it could 
be appropriately optimized.



-Travis

Re: [Numpy-discussion] std(axis=1) memory footprint issues + moving avg / stddev

[Rob H. <ro...@ho...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Torgil Svensson wrote:

> My original problem was to get an moving average and a moving standard
> deviation (120k rows and N=1000). For average I guess convolve should
> perform good, but is there anything smart for std()? For now I use ...
> 
>>>> moving_std=array([a[i:i+n].std() for i in range(len(a)-n)])
> 
> which seems to perform quite well.

You can always look for more fancy and unreadable solutions, but since
this one has the inner loop with a reasonable vector length (1000) coded
 in C, one can guess that the performance will be reasonable. I would
start looking for alternatives only if N drops significantly, say to <50.

Rob

- --
Rob W.W. Hooft  ||  ro...@ho...  ||  http://www.hooft.net/people/rob/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE8T/QH7J/Cv8rb3QRAtutAKCikJ1qLbedU4pNl7ZohHCLEAWVKACgji9R
6evNgk6R68/JnimUs4OOd98=
=htbE
-----END PGP SIGNATURE-----