Re: [Pytables-users] What is the best way to store date and timezone aware datetime objects in PyTables?

[Aquil H. A. <aqu...@gm...>]

Thanks. I think that that will do it!

On Tue, Jun 19, 2012 at 1:11 PM, Anthony Scopatz <sc...@gm...> wrote:

> On Tue, Jun 19, 2012 at 11:07 AM, Aquil H. Abdullah <
> aqu...@gm...> wrote:
>
>> Hello Anthony,
>>
>> Thanks for your reply. I did a little bit more searching and found the
>> discussion datetime objects on tables (
>> http://comments.gmane.org/gmane.comp.python.pytables.user/2469) ,which
>> is a bit more recent. In it you list your order of preference for storing
>> timestamps:
>>
>> 1. Create a numpy dtype or PyTables description that matches the
>> structure of datetime to your desired precision and save them in a Table.
>> If you want to save timezone information as well, I might add an extra
>> length-3 string column and save the str representation of the tzinfo field
>> ('UTC', 'KST', 'EDT').
>>
>> [Could you give an example?]
>>
>> The best I could come up with so far is to create a column of type
>> tables.Time32Col() and then transform my datetime object into a timestamp
>> and store it. For example,
>>
>> In [1]: import tables, pytz
>> In [2]: from datetime import datetime, timedelta
>> In [3]: from time import mktime
>> In [4]: import calendar
>> In [5]: dt = datetime.utcnow().replace(tzinfo=pytz.UTC)
>> In [6]: dt
>> Out[6]: datetime.datetime(2012, 6, 19, 15, 22, 19, 892159, tzinfo=<UTC>)
>> In [7]: ts = calendar.timegm(dt.timetuple())
>> In [8]: ts
>> Out[8]: 1340119339
>>
>> I can then store ts in my table, and retrieve it with
>>
>> In [35]: datetime.utcfromtimestamp(tbl.cols.timestamp[0])
>> Out[35]: datetime.datetime(2012, 6, 19, 15, 22, 19)
>>
>> although, at this point I've lost the timezone.
>>
>> Anyway, I am a little bit confused, because I don't understand the
>> difference between a Time32Col() and a Int32Col() in the example that I've
>> written. Do you have a better example?
>>
>
> So in terms of how they are stored on disk, Time32Col and Int32Col  are
> basically the same thing.  The difference is that they are flagged as being
> times so that other processes will know to interpret them as time stamps
> rather than plain old ints.  The same thing goes for Time64Col and
> Float64Col.
>
> Now as an example, to capture the full information you want, you will
> effectively have to store two columns: one for the time stamp and one for
> the time zone.  So you description will look like:
>
> desc = {'timestamp': Time64Col(pos=1), 'tz':  StringCol(3, pos=2)}
>
> Then the entries will look like tuples which match the desc like the
> following:
>
> In [35]: n = datetime.utcnow().replace(tzinfo=pytz.timezone('US/Central'))
>
> In [36]: time.mktime(n.timetuple()) + n.microsecond * 1e-6, str(n.tzname())
> Out[36]: (1340147092.873432, 'CST')
>
> Basically, you have to store the tz info next to the timestamp.  I hope
> this helps!
>
> Be Well
> Anthony
>
>
>>
>> Thanks!
>>
>>
>>
>> On Tue, Jun 19, 2012 at 2:16 AM, Anthony Scopatz <sc...@gm...>wrote:
>>
>>> Hey Aquil,
>>>
>>> Yes, the string method certainly works.  The other thing you could do
>>> that isn't mentioned in that post is have a table or 2D array whose first
>>> column is the float timestamp [1] and whose second column is a string
>>> repr of just the timezone name (or a int or float of the offset in sec of
>>> the timezone).  This will likely be faster.
>>>
>>> All of the strategies mentioned will work, but will have varying speeds.
>>>  I personally prefer anything with a timestamp since it relies on the
>>> canonical form.
>>>
>>> Be Well
>>> Anthony
>>>
>>> PS I am sorry that you have to deal with timezones.  They are a real
>>> pain!
>>>
>>> 1. http://pytables.github.com/usersguide/libref.html#tables.Time64Col
>>>
>>> On Mon, Jun 18, 2012 at 10:05 PM, Aquil H. Abdullah <
>>> aqu...@gm...> wrote:
>>>
>>>> I need to store dates and timezone aware datetimes in a PyTable.
>>>> Currently, I am storing values of those types as ISO 8601 strings. I ran
>>>> across the thread pytables for timeseries data (
>>>> http://osdir.com/ml/python.pytables.user/2007-11/msg00036.html) which
>>>> leads me to believe that what I am doing is the best way to store these
>>>> types of values,  but I just wanted to check in case I am missing something.
>>>>
>>>> Regards,
>>>>
>>>> --
>>>> Aquil H. Abdullah
>>>> aqu...@gm...
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Pytables-users mailing list
>>>> Pyt...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> --
>> Aquil H. Abdullah
>> aqu...@gm...
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>


-- 
Aquil H. Abdullah
aqu...@gm...