numpy-discussion

[Numpy-discussion] bad generator behaviour with sum

[<lis...@ma...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It seems like numpy.sum breaks generator expressions:

In [1]: sum(i*i for i in range(10))
Out[1]: 285

In [2]: from numpy import sum

In [3]: sum(i*i for i in range(10))
Out[3]: <generator object at 0x10eca58>

Is this intentional? If so, how do I get the behaviour that I am after?

Thanks,
C.

- --
Christopher Fonnesbeck
+ Atlanta, GA
+ fonnesbeck at mac.com
+ Contact me on AOL IM using email address


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFE8fEKkeka2iCbE4wRAoi6AKCjqJHodGOme56nohrG3X/njjaHgACeIkyn
PPB2+plZOyqV+HyLJgO+sSw=
=Y0wt
-----END PGP SIGNATURE-----

Re: [Numpy-discussion] bad generator behaviour with sum

[Charles R H. <cha...@gm...>]

Hi,

On 8/27/06, lis...@ma... <lis...@ma...> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> It seems like numpy.sum breaks generator expressions:
>
> In [1]: sum(i*i for i in range(10))
> Out[1]: 285
>
> In [2]: from numpy import sum
>
> In [3]: sum(i*i for i in range(10))
> Out[3]: <generator object at 0x10eca58>
>
> Is this intentional? If so, how do I get the behaviour that I am after?
>


In [3]: sum([i*i for i in range(10)])
Out[3]: 285

Chuck

Re: [Numpy-discussion] bad generator behaviour with sum

[Charles R H. <cha...@gm...>]

Hi Christopher,

On 8/27/06, Charles R Harris <cha...@gm...> wrote:
>
> Hi,
>
> On 8/27/06, lis...@ma... <lis...@ma...> wrote:
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > It seems like numpy.sum breaks generator expressions:
> >
> > In [1]: sum(i*i for i in range(10))
> > Out[1]: 285
> >
> > In [2]: from numpy import sum
> >
> > In [3]: sum(i*i for i in range(10))
> > Out[3]: <generator object at 0x10eca58>
> >
> > Is this intentional? If so, how do I get the behaviour that I am after?
> >
>
>
> In [3]: sum([i*i for i in range(10)])
> Out[3]: 285
>
> Chuck
>

The numarray.sum also fails to accept a generator as an argument. Because
python does and the imported sum overwrites it, we should probably check the
argument type and make it do the right thing.

Chuck

Re: [Numpy-discussion] bad generator behaviour with sum

[Charles R H. <cha...@gm...>]

Hi,

The problem seems to arise in the array constructor, which treats the
generator as a python object and creates an array containing that object.
So, do we want the possibility of an array of generators or should we
interpret it as a sort of list? I vote for that latter.

Chuck

On 8/27/06, Charles R Harris <cha...@gm...> wrote:
>
> Hi Christopher,
>
> On 8/27/06, Charles R Harris <cha...@gm...> wrote:
> >
> > Hi,
> >
> > On 8/27/06, lis...@ma... <lis...@ma...> wrote:
> > >
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > It seems like numpy.sum breaks generator expressions:
> > >
> > > In [1]: sum(i*i for i in range(10))
> > > Out[1]: 285
> > >
> > > In [2]: from numpy import sum
> > >
> > > In [3]: sum(i*i for i in range(10))
> > > Out[3]: <generator object at 0x10eca58>
> > >
> > > Is this intentional? If so, how do I get the behaviour that I am
> > > after?
> > >
> >
> >
> > In [3]: sum([i*i for i in range(10)])
> > Out[3]: 285
> >
> > Chuck
> >
>
> The numarray.sum also fails to accept a generator as an argument. Because
> python does and the imported sum overwrites it, we should probably check the
> argument type and make it do the right thing.
>
> Chuck
>
>
>

Re: [Numpy-discussion] bad generator behaviour with sum

[Tom D. <tom...@al...>]

I was thinking about this in the context of Giudo's comments at scipy
2006 that much of the language is moving away from lists toward
iterators.  He gave the keys of a dict as an example.

Numpy treats iterators, generators, etc as 0x0 PyObjects rather than
lazy generators of n dimensional data.  I guess my question for Travis
(any others much more expert than I in numpy) is is this intentional
or is it something that was never implemented because of the obvious
subtlties of defiing the correct semantics to make this work.

Personally i find it no big deal to use array(list(iter)) in the 1d
case and the list function combined with a list comprehension for the
2d case.  I usually know how many dimensions i expect so i find this
easy and i know about this peculiar behavior.  I find, however, that
this behavior is very suprising and confusing to the new user and i
don't usually have a good justification for it to answer them.

The ideal semantics, in my mind, would be if an iterator of iterators
of iterators, etc was no different in numpy than a list of lists of
lists, etc.  But I have no doubt that there are subtleties i am not
considering. Has anyone more familiar than I with the bowels of numpy
thought about this problem and see reasons why this is a bad idea or
just prohibitively difficult to implement?

On 8/27/06, Charles R Harris <cha...@gm...> wrote:
> Hi,
>
> The problem seems to arise in the array constructor, which treats the
> generator as a python object and creates an array containing that object.
> So, do we want the possibility of an array of generators or should we
> interpret it as a sort of list? I vote for that latter.
>
> Chuck
>
>
> On 8/27/06, Charles R Harris <cha...@gm...> wrote:
> >
> > Hi Christopher,
> >
> >
> >
> > On 8/27/06, Charles R Harris < cha...@gm...> wrote:
> > >
> > > Hi,
> > >
> > >
> > >
> > > On 8/27/06, lis...@ma... <lis...@ma...> wrote:
> > > > -----BEGIN PGP SIGNED MESSAGE-----
> > > > Hash: SHA1
> > > >
> > > > It seems like numpy.sum breaks generator expressions:
> > > >
> > > > In [1]: sum(i*i for i in range(10))
> > > > Out[1]: 285
> > > >
> > > > In [2]: from numpy import sum
> > > >
> > > > In [3]: sum(i*i for i in range(10))
> > > > Out[3]: <generator object at 0x10eca58>
> > > >
> > > > Is this intentional? If so, how do I get the behaviour that I am
> after?
> > > >
> > >
> > >
> > >
> > >
> > >
> > > In [3]: sum([i*i for i in range(10)])
> > >
> > > Out[3]: 285
> > >
> > > Chuck
> >
> >
> >
> > The numarray.sum also fails to accept a generator as an argument. Because
> python does and the imported sum overwrites it, we should probably check the
> argument type and make it do the right thing.
> >
> > Chuck
> >
> >
> >
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>

Re: [Numpy-discussion] bad generator behaviour with sum

[Tim H. <tim...@ie...>]

Tom Denniston wrote:
> I was thinking about this in the context of Giudo's comments at scipy
> 2006 that much of the language is moving away from lists toward
> iterators.  He gave the keys of a dict as an example.
>
> Numpy treats iterators, generators, etc as 0x0 PyObjects rather than
> lazy generators of n dimensional data.  I guess my question for Travis
> (any others much more expert than I in numpy) is is this intentional
> or is it something that was never implemented because of the obvious
> subtlties of defiing the correct semantics to make this work.
>   
More the latter than the former.

> Personally i find it no big deal to use array(list(iter)) in the 1d
> case and the list function combined with a list comprehension for the
> 2d case. 
There is a relatively new function fromiter, that materialized the last 
time this discussion came up that covers the above case. For example:
    numpy.fromiter((i*i for i in range(10)), int)

>  I usually know how many dimensions i expect so i find this
> easy and i know about this peculiar behavior.  I find, however, that
> this behavior is very suprising and confusing to the new user and i
> don't usually have a good justification for it to answer them.
>
> The ideal semantics, in my mind, would be if an iterator of iterators
> of iterators, etc was no different in numpy than a list of lists of
> lists, etc.  But I have no doubt that there are subtleties i am not
> considering. Has anyone more familiar than I with the bowels of numpy
> thought about this problem and see reasons why this is a bad idea or
> just prohibitively difficult to implement?
>   
There was some discussion about this several months ago and I even set 
out to implement it. I realized after not too long however that a 
complete solution, as you describe above, was going to be difficult and 
that I only really cared about the 1D case anyway, so punted and 
implemented fromiter instead. As I recall, there are two issues that 
complicate the general case:

   1. You need to specify the type or you gain no advantage over just
      instantiating the list. This is because you need to know the type
      before you allocate space for the array. Normally you do this by
      traversing the structure and looking at the contents. However for
      an iterable, you have to stash the results when you iterate over
      it looking for the type. This means that unless the array type is
      specified up front, you might as well just convert everything to
      lists as far as performance goes.
   2. For 1D arrays you can get away without knowing the shape by doing
      doing successive overallocation of memory, similar to the way list
      and array.array work. This is what fromiter does. I suppose the
      same tactic would work for iterators of iterators, but the
      bookkeeping becomes quite daunting.


Issue 1 is the real killer -- because of that a solution would either 
sometimes (mysteriously for the unitiated) be really inefficient or one 
would be required to specify types for array(iterable). The latter is my 
preference, but I'm beginning to think it would actually be better to 
always have to specify types. It's tempting to take another stab at 
this, in Python this time, and see if I can get a Python level soltuion 
working. However I don't have the time to try it right now.

-tim

> On 8/27/06, Charles R Harris <cha...@gm...> wrote:
>   
>> Hi,
>>
>> The problem seems to arise in the array constructor, which treats the
>> generator as a python object and creates an array containing that object.
>> So, do we want the possibility of an array of generators or should we
>> interpret it as a sort of list? I vote for that latter.
>>
>> Chuck
>>
>>
>> On 8/27/06, Charles R Harris <cha...@gm...> wrote:
>>     
>>> Hi Christopher,
>>>
>>>
>>>
>>> On 8/27/06, Charles R Harris < cha...@gm...> wrote:
>>>       
>>>> Hi,
>>>>
>>>>
>>>>
>>>> On 8/27/06, lis...@ma... <lis...@ma...> wrote:
>>>>         
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA1
>>>>>
>>>>> It seems like numpy.sum breaks generator expressions:
>>>>>
>>>>> In [1]: sum(i*i for i in range(10))
>>>>> Out[1]: 285
>>>>>
>>>>> In [2]: from numpy import sum
>>>>>
>>>>> In [3]: sum(i*i for i in range(10))
>>>>> Out[3]: <generator object at 0x10eca58>
>>>>>
>>>>> Is this intentional? If so, how do I get the behaviour that I am
>>>>>           
>> after?
>>     
>>>>
>>>>
>>>>
>>>> In [3]: sum([i*i for i in range(10)])
>>>>
>>>> Out[3]: 285
>>>>
>>>> Chuck
>>>>         
>>>
>>> The numarray.sum also fails to accept a generator as an argument. Because
>>>       
>> python does and the imported sum overwrites it, we should probably check the
>> argument type and make it do the right thing.
>>     
>>> Chuck
>>>
>>>
>>>
>>>       
>> -------------------------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job
>> easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Num...@li...
>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>
>>
>>
>>     
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>   

Re: [Numpy-discussion] bad generator behaviour with sum

[Travis O. <oli...@ie...>]

Tom Denniston wrote:
> I was thinking about this in the context of Giudo's comments at scipy
> 2006 that much of the language is moving away from lists toward
> iterators.  He gave the keys of a dict as an example.
>
> Numpy treats iterators, generators, etc as 0x0 PyObjects rather than
> lazy generators of n dimensional data.  I guess my question for Travis
> (any others much more expert than I in numpy) is is this intentional
> or is it something that was never implemented because of the obvious
> subtlties of defiing the correct semantics to make this work.
>
>   
It's not intentional, it's just that iterators came later and I did not 
try to figure out how to "do the right thing" in the array function.    
Thanks to Tim Hochberg, there is a separate  fromiter function that 
creates arrays from iterators.

> Personally i find it no big deal to use array(list(iter)) in the 1d
> case and the list function combined with a list comprehension for the
> 2d case.  I usually know how many dimensions i expect so i find this
> easy and i know about this peculiar behavior.  I find, however, that
> this behavior is very suprising and confusing to the new user and i
> don't usually have a good justification for it to answer them.
>   
The problem is that NumPy arrays need to know both how big they are and 
what data-type they are.  With iterators you have to basically construct 
the whole thing before you can even interrogate that question.   
Iterators were not part of the language when Numeric (from which NumPy 
got it's code base) was created.
> The ideal semantics, in my mind, would be if an iterator of iterators
> of iterators, etc was no different in numpy than a list of lists of
> lists, etc.  But I have no doubt that there are subtleties i am not
> considering. Has anyone more familiar than I with the bowels of numpy
> thought about this problem and see reasons why this is a bad idea or
> just prohibitively difficult to implement?
>   
It's been discussed before and ideas have been considered.   Right now, 
the fromiter function carries the load.   Whether or not to bring that 
functionality into the array function itself has been met with hesitancy 
because of how bulky the array function already is.

-Travis