[Pytables-users] Help in redoing current pytables schema.

[Jacob B. <jac...@gm...>]

Hello Pytables Users,

I am currently having a maximum number of children error within pytables. I
am trying to store stock updates within hdf5. My current schema is to have
one file represent a trading day, each table represent a particular
instrumentID (stock id) and have each record in the table belong to a
specific update with a timestamp (where the timestamp could be considered a
primary key).

I am currently having all tables be direct descendants of root.

The problem with this is that per day I have the following stats:

#of tables ::= 20000
#of Records per table ::= 250000

The problem persists in that 20000 is too many children to be associated
with a particular node. Continuing with this schema will consume
an exorbitant amount of memory and lead to slower query times.

Is there a way to redesign this schema so that it could work better with
pytables? Or is this simply too much data? Would it help to follow with the
current schema and just increase the depth of the tree by taking parts of
the instrumentId (instrumentId is an int64) as nodes?

Thanks,
Jacob

-- 
Jacob Bennett
Massachusetts Institute of Technology
Department of Electrical Engineering and Computer Science
Class of 2014| ben...@mi...