Re: [Pytables-users] Use of recarrays as representation for Tables in memory

[Anthony S. <sc...@gm...>]

That is reason enough for me really.  If someone really wants a recarray,
they could always convert an ndarray to this.  I think it is still worth
asking the numpy list what the status is...

Be Well
Anthony

On Thu, Jun 28, 2012 at 4:13 PM, Josh Ayers <jos...@gm...> wrote:

> There is a big difference in speed when iterating over the rows.  Possibly
> that was the reason structured arrays were chosen?  The issue is mentioned
> here: http://www.scipy.org/Cookbook/Recarray
>
> In a simple test, I get a difference of about 15x, so it is significant.
> Iterating over a recarray with 100,000 rows takes 0.22s, versus 0.014s for
> the structured array.  Here's the code.
>
> import time
> import numpy as np
>
> dtype = np.format_parser(['i4', 'i4'], [], [])
> N = 100000
> rec = np.recarray((N, ), dtype=dtype)
> struc = np.zeros((N, ), dtype=dtype)
>
> t1 = time.clock()
> for row in rec:
>     pass
> print time.clock() - t1
>
> t1 = time.clock()
> for row in struc:
>     pass
> print time.clock() - t1
>
>
>
>
> On Thu, Jun 28, 2012 at 1:31 PM, Anthony Scopatz <sc...@gm...>wrote:
>
>> On Thu, Jun 28, 2012 at 3:23 PM, Francesc Alted <fa...@py...>wrote:
>>
>>>  Yes, I think it would make more sense to return a recarray too.
>>> However, I remember many time ago (3, 4 years?) that NumPy developers were
>>> recommending using structured arrays instead of recarrays.  I don't
>>> remember exactly the arguments, but I think that was the reason why the
>>> structured arrays were declared the default for reading tables.  But this
>>> could be changed, of course...
>>>
>>
>> I remember this too Francesc.  I don't think that this has changed, but I
>> forgot the reasons.  Maybe I'll write to the numpy list later tonight,
>> unless someone else wants to...
>>
>>
>>>
>>> Francesc
>>>
>>>
>>> On 6/28/12 8:25 PM, Anthony Scopatz wrote:
>>>
>>> Hmmm Ok.  Maybe there needs to be a recarray flavor.
>>>
>>>  I kind of like just returning a normal ndarray, though I see your
>>> argument for returning a recarray.  Maybe some of the other devs can jump
>>> in here with an opinion.
>>>
>>>  Be Well
>>> Anthony
>>>
>>> On Thu, Jun 28, 2012 at 12:37 PM, Alvaro Tejero Cantero <al...@mi...
>>> > wrote:
>>>
>>>> I just tested: passing an object of type numpy.core.records.recarray
>>>> to the constructor of createTable and then reading back it into memory
>>>> via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray.
>>>>
>>>> Best,
>>>>
>>>> -á.
>>>>
>>>>
>>>> On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz <sc...@gm...>
>>>> wrote:
>>>> > Hi Alvaro,
>>>> >
>>>> > I think if you save the table as a record array, it should return you
>>>> a
>>>> > record array.  Or does it return a structured array?  Have you tried
>>>> this?
>>>> >
>>>> > Be Well
>>>> > Anthony
>>>> >
>>>> > On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero <
>>>> al...@mi...>
>>>> > wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> I've noticed that tables are loaded in memory as structured arrays.
>>>> >>
>>>> >> It seems that returning recarrays by default would be much in the
>>>> >> spirit of the natural naming preferences of PyTables.
>>>> >>
>>>> >> Is there a reason not to do so?
>>>> >>
>>>> >> Cheers,
>>>> >>
>>>> >> Álvaro.
>>>> >>
>>>> >>
>>>> >>
>>>> ------------------------------------------------------------------------------
>>>> >> Live Security Virtual Conference
>>>> >> Exclusive live event will cover all the ways today's security and
>>>> >> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> >> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> >> _______________________________________________
>>>> >> Pytables-users mailing list
>>>> >> Pyt...@li...
>>>> >> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>> >
>>>> >
>>>> >
>>>> >
>>>> ------------------------------------------------------------------------------
>>>> > Live Security Virtual Conference
>>>> > Exclusive live event will cover all the ways today's security and
>>>> > threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> > will include endpoint security, mobile security and the latest in
>>>> malware
>>>> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> > _______________________________________________
>>>> > Pytables-users mailing list
>>>> > Pyt...@li...
>>>> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>> >
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Pytables-users mailing list
>>>> Pyt...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>
>>>
>>>
>>> _______________________________________________
>>> Pytables-users mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>>
>>> --
>>> Francesc Alted
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>