pytables-users

[Pytables-users] Large (to very large) datasets...

[Andrea G. <and...@gm...>]

Hi All,

    I am pretty new to pytables and I am facing a problem of actually
storing and retrieving data to/from a large dataset. My situation is
the following:

1. I am running stochastic simulations of a number of objects
(typically between 100-1,000 simulations);
2. For every simulation, I have around 1,200 "objects", and for each
of them I have 7 timeseries of 600 time-steps each.

I thought of using pytables to try and get some sense out of my
simulations, but I am failing to implement something intelligent (or
fast, which is important as well...).

The attached script (modified from the pytables tutorial) does the following:

1. Create a table containing these "objects";
2. Adds 1,200 rows, one per "object": for each "object", I assign a 3D
array defined as:

results = Float32Col(shape=(NUM_SIM, len(ALL_DATES), 7))

Where NUM_SIM is the number of simulations and ALL_DATES are the timesteps.

3. For every simulation, I update the "object" results (using random
numbers in the script).

The timings on my computer are as follows (in seconds):

H5 file creation time: 22.510

Saving results for simulation 1   : 3.33599996567
Saving results for simulation 2   : 6.2429997921
Saving results for simulation 3   : 9.15199995041
Saving results for simulation 4   : 12.0759999752
Saving results for simulation 5   : 15.2199997902
Saving results for simulation 6   : 17.9159998894
Saving results for simulation 7   : 21.0659999847
Saving results for simulation 8   : 23.6459999084
Saving results for simulation 9   : 26.5359997749
Saving results for simulation 10  : 29.5579998493

As you can see, at every simulation the processing time increases by 3
seconds, so by the time I get to 100 or 1,000 I will have more than
enough time for 15 coffees in the morning :-D
Also, the file creation time is somewhat on the slow side...

I am sure I am missing a lot of things here, so I would appreciate any
suggestion to implement my code in a better/more intelligent way (and
also suggestions on other approaches in order to do what I am trying
to do).

Thank you in advance for your suggestions.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://www.infinity77.net

Re: [Pytables-users] Large (to very large) datasets...

[Andrea G. <and...@gm...>]

Hi Anthony,

On 30 October 2012 22:52, Anthony Scopatz wrote:
> Hi Andrea,
>
> Your problem is two fold.
>
> 1. Your timing wasn't reporting the time per data set, but rather the total
> time since writing all data sets.  You need to put the start time in the
> loop to get the time per data set.
>
> 2. Your larger problem was that you were writing too many times.  Generally
> it is faster to write fewer, bigger sets of data than performing a lot of
> small write operations.  Since you had data set opening and writing in a
> doubly nested loop, it is not surprising that you were getting terrible
> performance.   You were basically maximizing HDF5 overhead ;).  Using
> slicing I removed the outermost loop and saw timings like the following:
>
> H5 file creation time: 7.406
>
> Saving results for table: 0.0105440616608
> Saving results for table: 0.0158948898315
> Saving results for table: 0.0164661407471
> Saving results for table: 0.00654292106628
> Saving results for table: 0.00676298141479
> Saving results for table: 0.00664114952087
> Saving results for table: 0.0066990852356
> Saving results for table: 0.00687289237976
> Saving results for table: 0.00664210319519
> Saving results for table: 0.0157809257507
> Saving results for table: 0.0141618251801
> Saving results for table: 0.00796294212341
>
> Please see the attached version, at around line 82.  Additionally, if you
> need to focus on performance I would recommend reading the following
> (http://pytables.github.com/usersguide/optimization.html).  PyTables can be
> blazingly fast when implemented correctly.  I would highly recommend looking
> into compression.
>
> I hope this helps!

Thank you for your answer; indeed, I was timing it wrongly (I really
need to go to sleep...). However, although I understand the need of
"writing fewer", I am not sure I can actually do it in my situations.
Let me explain:

1. I have a GUI which starts a number of parallel processes (up to 16,
depending on a user selection);
2. These processes actually do the computation/simulations - so, if I
have 1,000 simulations to run and 8 parallel processes, each process
gets 125 simulations (each of which holds 1,200 "objects" with a 600x7
timeseries matrix per object).

If I had to write out the results only at the end, it would mean for
me to find a way to share the 1,200 "objects" matrices in all the
parallel processes (and I am not sure if pytables is going to complain
when multiple concurrent processes try to access the same underlying
HDF5 file).

Or I could create one HDF file per process, but given the nature of
the simulation I am running, every "object" in the 1,200 "objects"
pool would need to keep a reference to a 125x600x7 matrix (assuming
1,000 simulations and 8 processes) around in memory *OR* I will need
to write the results to the HDF5 file for every simulation. Although
we have extremely powerful PCs at work, I am not sure it is the right
way to go...

As always, I am open to all suggestions on how to improve my approach.

Thank you again for your quick and enlightening answer.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://www.infinity77.net

# ------------------------------------------------------------- #
def ask_mailing_list_support(email):

    if mention_platform_and_version() and include_sample_app():
        send_message(email)
    else:
        install_malware()
        erase_hard_drives()
# ------------------------------------------------------------- #

Re: [Pytables-users] Large (to very large) datasets...

[Anthony S. <sc...@gm...>]

On Tue, Oct 30, 2012 at 6:20 PM, Andrea Gavana <and...@gm...>wrote:

> Hi Anthony,
>
> On 30 October 2012 22:52, Anthony Scopatz wrote:
> > Hi Andrea,
> >
> > Your problem is two fold.
> >
> > 1. Your timing wasn't reporting the time per data set, but rather the
> total
> > time since writing all data sets.  You need to put the start time in the
> > loop to get the time per data set.
> >
> > 2. Your larger problem was that you were writing too many times.
>  Generally
> > it is faster to write fewer, bigger sets of data than performing a lot of
> > small write operations.  Since you had data set opening and writing in a
> > doubly nested loop, it is not surprising that you were getting terrible
> > performance.   You were basically maximizing HDF5 overhead ;).  Using
> > slicing I removed the outermost loop and saw timings like the following:
> >
> > H5 file creation time: 7.406
> >
> > Saving results for table: 0.0105440616608
> > Saving results for table: 0.0158948898315
> > Saving results for table: 0.0164661407471
> > Saving results for table: 0.00654292106628
> > Saving results for table: 0.00676298141479
> > Saving results for table: 0.00664114952087
> > Saving results for table: 0.0066990852356
> > Saving results for table: 0.00687289237976
> > Saving results for table: 0.00664210319519
> > Saving results for table: 0.0157809257507
> > Saving results for table: 0.0141618251801
> > Saving results for table: 0.00796294212341
> >
> > Please see the attached version, at around line 82.  Additionally, if you
> > need to focus on performance I would recommend reading the following
> > (http://pytables.github.com/usersguide/optimization.html).  PyTables
> can be
> > blazingly fast when implemented correctly.  I would highly recommend
> looking
> > into compression.
> >
> > I hope this helps!
>
> Thank you for your answer; indeed, I was timing it wrongly (I really
> need to go to sleep...). However, although I understand the need of
> "writing fewer", I am not sure I can actually do it in my situations.
> Let me explain:
>
> 1. I have a GUI which starts a number of parallel processes (up to 16,
> depending on a user selection);
> 2. These processes actually do the computation/simulations - so, if I
> have 1,000 simulations to run and 8 parallel processes, each process
> gets 125 simulations (each of which holds 1,200 "objects" with a 600x7
> timeseries matrix per object).
>

Well, you can at least change the order of the loops and see if that helps.
That is rather than doing:

for i in xrange():
    for p in table:

Do the following instead:

for p in table:
    for i in xrange():

I don't believe that this will help too much since you are still writing
every element individually..


>
> If I had to write out the results only at the end, it would mean for
> me to find a way to share the 1,200 "objects" matrices in all the
> parallel processes (and I am not sure if pytables is going to complain
> when multiple concurrent processes try to access the same underlying
> HDF5 file).
>

Reading in parallel works pretty well.  Writing causes more headaches
but can be done.


> Or I could create one HDF file per process, but given the nature of
> the simulation I am running, every "object" in the 1,200 "objects"
> pool would need to keep a reference to a 125x600x7 matrix (assuming
> 1,000 simulations and 8 processes) around in memory *OR* I will need
> to write the results to the HDF5 file for every simulation. Although
> we have extremely powerful PCs at work, I am not sure it is the right
> way to go...
>
> As always, I am open to all suggestions on how to improve my approach.
>

My basic suggestion is to have all of you processes produce results which
are then
aggregated by a single master process.  This master is the only one which
has write
access to the hdf5 file and will allow you to create larger arrays and
minimize the
number of writes that you do.

You'll probably want to take a look at this example:
https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py

I think that there might be a page in the docs about it now too...

But I think that this is the strategy that you want to pursue.  Multiple
compute processes, one write process.


>
> Thank you again for your quick and enlightening answer.
>

No problem!
Be Well
Anthony



>
> Andrea.
>
> "Imagination Is The Only Weapon In The War Against Reality."
> http://www.infinity77.net
>
> # ------------------------------------------------------------- #
> def ask_mailing_list_support(email):
>
>     if mention_platform_and_version() and include_sample_app():
>         send_message(email)
>     else:
>         install_malware()
>         erase_hard_drives()
> # ------------------------------------------------------------- #
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

Re: [Pytables-users] Large (to very large) datasets...

[Andrea G. <and...@gm...>]

Hi Anthony & All,

On 30 October 2012 23:31, Anthony Scopatz wrote:
> On Tue, Oct 30, 2012 at 6:20 PM, Andrea Gavana <and...@gm...>
> wrote:
>>
>> Hi Anthony,
>>
>> On 30 October 2012 22:52, Anthony Scopatz wrote:
>> > Hi Andrea,
>> >
>> > Your problem is two fold.
>> >
>> > 1. Your timing wasn't reporting the time per data set, but rather the
>> > total
>> > time since writing all data sets.  You need to put the start time in the
>> > loop to get the time per data set.
>> >
>> > 2. Your larger problem was that you were writing too many times.
>> > Generally
>> > it is faster to write fewer, bigger sets of data than performing a lot
>> > of
>> > small write operations.  Since you had data set opening and writing in a
>> > doubly nested loop, it is not surprising that you were getting terrible
>> > performance.   You were basically maximizing HDF5 overhead ;).  Using
>> > slicing I removed the outermost loop and saw timings like the following:
>> >
>> > H5 file creation time: 7.406
>> >
>> > Saving results for table: 0.0105440616608
>> > Saving results for table: 0.0158948898315
>> > Saving results for table: 0.0164661407471
>> > Saving results for table: 0.00654292106628
>> > Saving results for table: 0.00676298141479
>> > Saving results for table: 0.00664114952087
>> > Saving results for table: 0.0066990852356
>> > Saving results for table: 0.00687289237976
>> > Saving results for table: 0.00664210319519
>> > Saving results for table: 0.0157809257507
>> > Saving results for table: 0.0141618251801
>> > Saving results for table: 0.00796294212341
>> >
>> > Please see the attached version, at around line 82.  Additionally, if
>> > you
>> > need to focus on performance I would recommend reading the following
>> > (http://pytables.github.com/usersguide/optimization.html).  PyTables can
>> > be
>> > blazingly fast when implemented correctly.  I would highly recommend
>> > looking
>> > into compression.
>> >
>> > I hope this helps!
>>
>> Thank you for your answer; indeed, I was timing it wrongly (I really
>> need to go to sleep...). However, although I understand the need of
>> "writing fewer", I am not sure I can actually do it in my situations.
>> Let me explain:
>>
>> 1. I have a GUI which starts a number of parallel processes (up to 16,
>> depending on a user selection);
>> 2. These processes actually do the computation/simulations - so, if I
>> have 1,000 simulations to run and 8 parallel processes, each process
>> gets 125 simulations (each of which holds 1,200 "objects" with a 600x7
>> timeseries matrix per object).
>
>
> Well, you can at least change the order of the loops and see if that helps.
> That is rather than doing:
>
> for i in xrange():
>     for p in table:
>
> Do the following instead:
>
> for p in table:
>     for i in xrange():
>
> I don't believe that this will help too much since you are still writing
> every element individually..
>
>>
>>
>> If I had to write out the results only at the end, it would mean for
>> me to find a way to share the 1,200 "objects" matrices in all the
>> parallel processes (and I am not sure if pytables is going to complain
>> when multiple concurrent processes try to access the same underlying
>> HDF5 file).
>
>
> Reading in parallel works pretty well.  Writing causes more headaches
> but can be done.
>
>>
>> Or I could create one HDF file per process, but given the nature of
>> the simulation I am running, every "object" in the 1,200 "objects"
>> pool would need to keep a reference to a 125x600x7 matrix (assuming
>> 1,000 simulations and 8 processes) around in memory *OR* I will need
>> to write the results to the HDF5 file for every simulation. Although
>> we have extremely powerful PCs at work, I am not sure it is the right
>> way to go...
>>
>> As always, I am open to all suggestions on how to improve my approach.
>
>
> My basic suggestion is to have all of you processes produce results which
> are then
> aggregated by a single master process.  This master is the only one which
> has write
> access to the hdf5 file and will allow you to create larger arrays and
> minimize the
> number of writes that you do.
>
> You'll probably want to take a look at this example:
> https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py
>
> I think that there might be a page in the docs about it now too...
>
> But I think that this is the strategy that you want to pursue.  Multiple
> compute processes, one write process.

Thank you for all your suggestions. I managed to slightly modify the
script you attached and I am also experimenting with compression.
However, in the newly attached script the underlying table is not
modified, i.e., this assignment:

for p in table:
    p['results'][:NUM_SIM, :, :] = numpy.random.random(size=(NUM_SIM,
len(ALL_DATES), 7))
    table.flush()

Seems to be doing nothing (i.e., printing out the 'results' attribute
for an object class prints a matrix full of zeros instead of random
numbers...). Also, on my PC at work, the file creation time is
tremendously slow (76 seconds for a 100 simulations - 1.9 GB file).

In order to understand what's going on, I set back the number of
simulations to 10 (NUM_SIM=10), but still I am getting only zeros out
of the table. This is what my script is printing out:

H5 file creation time: 7.652

Saving results for table: 1.03400015831

Results (should be random...)

Object name   : KB0001
Object results:
[[[ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  ...,
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]]

 [[ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  ...,
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]]

 [[ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  ...,
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]]

 ...,
 [[ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  ...,
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]]

 [[ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  ...,
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]]

 [[ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  ...,
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]
  [ 0.  0.  0. ...,  0.  0.  0.]]]



I am on Windows Vista, Python 2.7.2 64-bit  from EPD 7.1-2, pytables
version '2.3b1.devpro'.

Any suggestion is really appreciated. Thank you in advance.


Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://www.infinity77.net

# ------------------------------------------------------------- #
def ask_mailing_list_support(email):

    if mention_platform_and_version() and include_sample_app():
        send_message(email)
    else:
        install_malware()
        erase_hard_drives()
# ------------------------------------------------------------- #

Re: [Pytables-users] Large (to very large) datasets...

[Francesc A. <fa...@gm...>]

On 10/31/12 4:30 AM, Andrea Gavana wrote:
> Thank you for all your suggestions. I managed to slightly modify the
> script you attached and I am also experimenting with compression.
> However, in the newly attached script the underlying table is not
> modified, i.e., this assignment:
>
> for p in table:
>      p['results'][:NUM_SIM, :, :] = numpy.random.random(size=(NUM_SIM,
> len(ALL_DATES), 7))
>      table.flush()

For modifying row values you need to assign a complete row object. 
Something like:

for i in range(len(table)):
     myrow = table[i]
     myrow['results'][:NUM_SIM, :, :] = 
numpy.random.random(size=(NUM_SIM, len(ALL_DATES), 7))
     table[i] = myrow

You may also use Table.modifyColumn() for better efficiency.  Look at 
the different modification methods here:

http://pytables.github.com/usersguide/libref/structured_storage.html#table-methods-writing

and experiment with them.

>
> Seems to be doing nothing (i.e., printing out the 'results' attribute
> for an object class prints a matrix full of zeros instead of random
> numbers...). Also, on my PC at work, the file creation time is
> tremendously slow (76 seconds for a 100 simulations - 1.9 GB file).
>
> In order to understand what's going on, I set back the number of
> simulations to 10 (NUM_SIM=10), but still I am getting only zeros out
> of the table. This is what my script is printing out:
>
> H5 file creation time: 7.652

Hmm, in my modest Core2 laptop I'm getting this:

H5 file creation time: 1.294

Also, by using compression with zlib level 1:

H5 file creation time: 1.900

And using blosc level 5:

H5 file creation time: 0.244

HTH,

-- 
Francesc Alted

Re: [Pytables-users] Large (to very large) datasets...

[Andrea G. <and...@gm...>]

Hi Francesc & All,

On 31 October 2012 14:13, Francesc Alted wrote:
> On 10/31/12 4:30 AM, Andrea Gavana wrote:
>> Thank you for all your suggestions. I managed to slightly modify the
>> script you attached and I am also experimenting with compression.
>> However, in the newly attached script the underlying table is not
>> modified, i.e., this assignment:
>>
>> for p in table:
>>      p['results'][:NUM_SIM, :, :] = numpy.random.random(size=(NUM_SIM,
>> len(ALL_DATES), 7))
>>      table.flush()
>
> For modifying row values you need to assign a complete row object.
> Something like:
>
> for i in range(len(table)):
>      myrow = table[i]
>      myrow['results'][:NUM_SIM, :, :] =
> numpy.random.random(size=(NUM_SIM, len(ALL_DATES), 7))
>      table[i] = myrow
>
> You may also use Table.modifyColumn() for better efficiency.  Look at
> the different modification methods here:
>
> http://pytables.github.com/usersguide/libref/structured_storage.html#table-methods-writing
>
> and experiment with them.

Thank you, I have tried different approaches and they all seem to run
more or less at the same speed (see below). I had to slightly modify
your code from:

table[i] = myrow

to

table[i] = [myrow]

To avoid exceptions.

In the newly attached file, I switched to blosc for compression (but
with compression level 1) and run a few sensitivities. By calling the
attached script as:

python pytables_test.py NUM_SIM

where "NUM_SIM" is an integer, I get the following timings and file sizes:

C:\MyProjects\Phaser\tests>python pytables_test.py 10
Number of simulations   : 10
H5 file creation time   : 0.879s
Saving results for table: 6.413s
H5 file size (MB)       : 193


C:\MyProjects\Phaser\tests>python pytables_test.py 100
Number of simulations   : 100
H5 file creation time   : 4.155s
Saving results for table: 86.326s
H5 file size (MB)       : 1935


I dont think I will try the 1,000 simulations case :-) . I believe I
still don't understand what the best strategy would be for my problem.
I basically need to save all the simulation results for all the 1,200
"objects", each of which has a timeseries matrix of 600x7 size. In the
GUI I have, these 1,200 "objects" are grouped into multiple
categories, and multiple categories can reference the same "object",
i.e.:

Category_1: object_1, object_23, object_543, etc...
Category_2: object_23, object_100, object_543, etc...

So my idea was to save all the "objects" results to disk and, upon the
user's choice, build the categories results "on the fly", i.e. by
seeking the H5 file on disk for the "objects" belonging to that
specific category and summing up all their results (over time, i.e.
the 600 time-steps). Maybe I would be better off with a 4D array
(NUM_OBJECTS, NUM_SIM, TSTEPS, 7) as a table, but then I will lose the
ability to reference the "objects" by their names...

I welcome in advance any suggestion on how to improve my thinking on
this matter. Thanks for all the answers I received.


Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://www.infinity77.net

# ------------------------------------------------------------- #
def ask_mailing_list_support(email):

    if mention_platform_and_version() and include_sample_app():
        send_message(email)
    else:
        install_malware()
        erase_hard_drives()
# ------------------------------------------------------------- #

Re: [Pytables-users] Large (to very large) datasets...

[Francesc A. <fa...@gm...>]

On 10/31/12 10:12 AM, Andrea Gavana wrote:
> Hi Francesc & All,
>
> On 31 October 2012 14:13, Francesc Alted wrote:
>> On 10/31/12 4:30 AM, Andrea Gavana wrote:
>>> Thank you for all your suggestions. I managed to slightly modify the
>>> script you attached and I am also experimenting with compression.
>>> However, in the newly attached script the underlying table is not
>>> modified, i.e., this assignment:
>>>
>>> for p in table:
>>>       p['results'][:NUM_SIM, :, :] = numpy.random.random(size=(NUM_SIM,
>>> len(ALL_DATES), 7))
>>>       table.flush()
>> For modifying row values you need to assign a complete row object.
>> Something like:
>>
>> for i in range(len(table)):
>>       myrow = table[i]
>>       myrow['results'][:NUM_SIM, :, :] =
>> numpy.random.random(size=(NUM_SIM, len(ALL_DATES), 7))
>>       table[i] = myrow
>>
>> You may also use Table.modifyColumn() for better efficiency.  Look at
>> the different modification methods here:
>>
>> http://pytables.github.com/usersguide/libref/structured_storage.html#table-methods-writing
>>
>> and experiment with them.
> Thank you, I have tried different approaches and they all seem to run
> more or less at the same speed (see below). I had to slightly modify
> your code from:
>
> table[i] = myrow
>
> to
>
> table[i] = [myrow]
>
> To avoid exceptions.
>
> In the newly attached file, I switched to blosc for compression (but
> with compression level 1) and run a few sensitivities. By calling the
> attached script as:
>
> python pytables_test.py NUM_SIM
>
> where "NUM_SIM" is an integer, I get the following timings and file sizes:
>
> C:\MyProjects\Phaser\tests>python pytables_test.py 10
> Number of simulations   : 10
> H5 file creation time   : 0.879s
> Saving results for table: 6.413s
> H5 file size (MB)       : 193
>
>
> C:\MyProjects\Phaser\tests>python pytables_test.py 100
> Number of simulations   : 100
> H5 file creation time   : 4.155s
> Saving results for table: 86.326s
> H5 file size (MB)       : 1935
>
>
> I dont think I will try the 1,000 simulations case :-) . I believe I
> still don't understand what the best strategy would be for my problem.
> I basically need to save all the simulation results for all the 1,200
> "objects", each of which has a timeseries matrix of 600x7 size. In the
> GUI I have, these 1,200 "objects" are grouped into multiple
> categories, and multiple categories can reference the same "object",
> i.e.:
>
> Category_1: object_1, object_23, object_543, etc...
> Category_2: object_23, object_100, object_543, etc...
>
> So my idea was to save all the "objects" results to disk and, upon the
> user's choice, build the categories results "on the fly", i.e. by
> seeking the H5 file on disk for the "objects" belonging to that
> specific category and summing up all their results (over time, i.e.
> the 600 time-steps). Maybe I would be better off with a 4D array
> (NUM_OBJECTS, NUM_SIM, TSTEPS, 7) as a table, but then I will lose the
> ability to reference the "objects" by their names...

You should keep trying experimenting with different approaches and 
discover the one that works for you the best.  Regarding using the 4D 
array as a table, I might be misunderstanding your problem, but you can 
still reference objects by name by using:

row = table.where("name == %s" % my_name)
table[row.nrow] = ...

You may want to index the 'name' column for better performance.

-- 
Francesc Alted

Re: [Pytables-users] Large (to very large) datasets...

[Andrea G. <and...@gm...>]

Hi Francesc and All,

On 31 October 2012 21:02, Francesc Alted wrote:
> On 10/31/12 10:12 AM, Andrea Gavana wrote:
>> Hi Francesc & All,
>>
>> On 31 October 2012 14:13, Francesc Alted wrote:
>>> On 10/31/12 4:30 AM, Andrea Gavana wrote:
>>>> Thank you for all your suggestions. I managed to slightly modify the
>>>> script you attached and I am also experimenting with compression.
>>>> However, in the newly attached script the underlying table is not
>>>> modified, i.e., this assignment:
>>>>
>>>> for p in table:
>>>>       p['results'][:NUM_SIM, :, :] = numpy.random.random(size=(NUM_SIM,
>>>> len(ALL_DATES), 7))
>>>>       table.flush()
>>> For modifying row values you need to assign a complete row object.
>>> Something like:
>>>
>>> for i in range(len(table)):
>>>       myrow = table[i]
>>>       myrow['results'][:NUM_SIM, :, :] =
>>> numpy.random.random(size=(NUM_SIM, len(ALL_DATES), 7))
>>>       table[i] = myrow
>>>
>>> You may also use Table.modifyColumn() for better efficiency.  Look at
>>> the different modification methods here:
>>>
>>> http://pytables.github.com/usersguide/libref/structured_storage.html#table-methods-writing
>>>
>>> and experiment with them.
>> Thank you, I have tried different approaches and they all seem to run
>> more or less at the same speed (see below). I had to slightly modify
>> your code from:
>>
>> table[i] = myrow
>>
>> to
>>
>> table[i] = [myrow]
>>
>> To avoid exceptions.
>>
>> In the newly attached file, I switched to blosc for compression (but
>> with compression level 1) and run a few sensitivities. By calling the
>> attached script as:
>>
>> python pytables_test.py NUM_SIM
>>
>> where "NUM_SIM" is an integer, I get the following timings and file sizes:
>>
>> C:\MyProjects\Phaser\tests>python pytables_test.py 10
>> Number of simulations   : 10
>> H5 file creation time   : 0.879s
>> Saving results for table: 6.413s
>> H5 file size (MB)       : 193
>>
>>
>> C:\MyProjects\Phaser\tests>python pytables_test.py 100
>> Number of simulations   : 100
>> H5 file creation time   : 4.155s
>> Saving results for table: 86.326s
>> H5 file size (MB)       : 1935
>>
>>
>> I dont think I will try the 1,000 simulations case :-) . I believe I
>> still don't understand what the best strategy would be for my problem.
>> I basically need to save all the simulation results for all the 1,200
>> "objects", each of which has a timeseries matrix of 600x7 size. In the
>> GUI I have, these 1,200 "objects" are grouped into multiple
>> categories, and multiple categories can reference the same "object",
>> i.e.:
>>
>> Category_1: object_1, object_23, object_543, etc...
>> Category_2: object_23, object_100, object_543, etc...
>>
>> So my idea was to save all the "objects" results to disk and, upon the
>> user's choice, build the categories results "on the fly", i.e. by
>> seeking the H5 file on disk for the "objects" belonging to that
>> specific category and summing up all their results (over time, i.e.
>> the 600 time-steps). Maybe I would be better off with a 4D array
>> (NUM_OBJECTS, NUM_SIM, TSTEPS, 7) as a table, but then I will lose the
>> ability to reference the "objects" by their names...
>
> You should keep trying experimenting with different approaches and
> discover the one that works for you the best.  Regarding using the 4D
> array as a table, I might be misunderstanding your problem, but you can
> still reference objects by name by using:
>
> row = table.where("name == %s" % my_name)
> table[row.nrow] = ...
>
> You may want to index the 'name' column for better performance.

I did spend quite some time experimenting today (actually almost the
whole day), but even the task of writing a 4D array (created via
createCArray) to disk is somehow overwhelming from a GUI point of
view. My 4D array is a 1,200x100x600x7, and on my PC at work (16
cores, 96 GB RAM, 3.0 GHz Windows Vista 64bit with PyTables pro) it
takes close to 80 seconds to populate it with random arrays and save
it to disk. This is almost a lifetime in the GUI world, and 100
simulation is possibly the simplest case I have.

As I said before, I am probably completely missing the point; the fact
that my script seems to be "un-improvable" in terms of speed is quite
a demonstration of it. But given the constraints (in terms of time and
GUI responsiveness) we have I will probably go back to my previous
approach of saving the simulation results for the higher level
categories only, discarding the ones for the 1,200 "objects".

I love the idea of being able to seek results in real-time via a HDF5
file on my drive and having all the simulation results readily
available, but the actual "saving" time is somehow a showstopper.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://www.infinity77.net

# ------------------------------------------------------------- #
def ask_mailing_list_support(email):

    if mention_platform_and_version() and include_sample_app():
        send_message(email)
    else:
        install_malware()
        erase_hard_drives()
# ------------------------------------------------------------- #

Re: [Pytables-users] Large (to very large) datasets...

[Francesc A. <fa...@gm...>]

On 10/31/12 4:02 PM, Francesc Alted wrote:
> On 10/31/12 10:12 AM, Andrea Gavana wrote:
>> Hi Francesc & All,
>>
>> On 31 October 2012 14:13, Francesc Alted wrote:
>>> On 10/31/12 4:30 AM, Andrea Gavana wrote:
>>>> Thank you for all your suggestions. I managed to slightly modify the
>>>> script you attached and I am also experimenting with compression.
>>>> However, in the newly attached script the underlying table is not
>>>> modified, i.e., this assignment:
>>>>
>>>> for p in table:
>>>>       p['results'][:NUM_SIM, :, :] = 
>>>> numpy.random.random(size=(NUM_SIM,
>>>> len(ALL_DATES), 7))
>>>>       table.flush()
>>> For modifying row values you need to assign a complete row object.
>>> Something like:
>>>
>>> for i in range(len(table)):
>>>       myrow = table[i]
>>>       myrow['results'][:NUM_SIM, :, :] =
>>> numpy.random.random(size=(NUM_SIM, len(ALL_DATES), 7))
>>>       table[i] = myrow
>>>
>>> You may also use Table.modifyColumn() for better efficiency. Look at
>>> the different modification methods here:
>>>
>>> http://pytables.github.com/usersguide/libref/structured_storage.html#table-methods-writing 
>>>
>>>
>>> and experiment with them.
>> Thank you, I have tried different approaches and they all seem to run
>> more or less at the same speed (see below). I had to slightly modify
>> your code from:
>>
>> table[i] = myrow
>>
>> to
>>
>> table[i] = [myrow]
>>
>> To avoid exceptions.
>>
>> In the newly attached file, I switched to blosc for compression (but
>> with compression level 1) and run a few sensitivities. By calling the
>> attached script as:
>>
>> python pytables_test.py NUM_SIM
>>
>> where "NUM_SIM" is an integer, I get the following timings and file 
>> sizes:
>>
>> C:\MyProjects\Phaser\tests>python pytables_test.py 10
>> Number of simulations   : 10
>> H5 file creation time   : 0.879s
>> Saving results for table: 6.413s
>> H5 file size (MB)       : 193
>>
>>
>> C:\MyProjects\Phaser\tests>python pytables_test.py 100
>> Number of simulations   : 100
>> H5 file creation time   : 4.155s
>> Saving results for table: 86.326s
>> H5 file size (MB)       : 1935
>>
>>
>> I dont think I will try the 1,000 simulations case :-) . I believe I
>> still don't understand what the best strategy would be for my problem.
>> I basically need to save all the simulation results for all the 1,200
>> "objects", each of which has a timeseries matrix of 600x7 size. In the
>> GUI I have, these 1,200 "objects" are grouped into multiple
>> categories, and multiple categories can reference the same "object",
>> i.e.:
>>
>> Category_1: object_1, object_23, object_543, etc...
>> Category_2: object_23, object_100, object_543, etc...
>>
>> So my idea was to save all the "objects" results to disk and, upon the
>> user's choice, build the categories results "on the fly", i.e. by
>> seeking the H5 file on disk for the "objects" belonging to that
>> specific category and summing up all their results (over time, i.e.
>> the 600 time-steps). Maybe I would be better off with a 4D array
>> (NUM_OBJECTS, NUM_SIM, TSTEPS, 7) as a table, but then I will lose the
>> ability to reference the "objects" by their names...
>
> You should keep trying experimenting with different approaches and 
> discover the one that works for you the best.  Regarding using the 4D 
> array as a table, I might be misunderstanding your problem, but you 
> can still reference objects by name by using:
>
> row = table.where("name == %s" % my_name)
> table[row.nrow] = ...

Uh, I rather meant:

row = table.readWhere("name == %s" % my_name)
table[row.nrow] = ...

but you probably got the idea already.

-- 
Francesc Alted

Re: [Pytables-users] Large (to very large) datasets...

[Francesc A. <fa...@gm...>]

On 10/31/12 4:05 PM, Francesc Alted wrote:
> On 10/31/12 4:02 PM, Francesc Alted wrote:
>> On 10/31/12 10:12 AM, Andrea Gavana wrote:
>>> Hi Francesc & All,
>>>
>>> On 31 October 2012 14:13, Francesc Alted wrote:
>>>> On 10/31/12 4:30 AM, Andrea Gavana wrote:
>>>>> Thank you for all your suggestions. I managed to slightly modify the
>>>>> script you attached and I am also experimenting with compression.
>>>>> However, in the newly attached script the underlying table is not
>>>>> modified, i.e., this assignment:
>>>>>
>>>>> for p in table:
>>>>>       p['results'][:NUM_SIM, :, :] = 
>>>>> numpy.random.random(size=(NUM_SIM,
>>>>> len(ALL_DATES), 7))
>>>>>       table.flush()
>>>> For modifying row values you need to assign a complete row object.
>>>> Something like:
>>>>
>>>> for i in range(len(table)):
>>>>       myrow = table[i]
>>>>       myrow['results'][:NUM_SIM, :, :] =
>>>> numpy.random.random(size=(NUM_SIM, len(ALL_DATES), 7))
>>>>       table[i] = myrow
>>>>
>>>> You may also use Table.modifyColumn() for better efficiency. Look at
>>>> the different modification methods here:
>>>>
>>>> http://pytables.github.com/usersguide/libref/structured_storage.html#table-methods-writing 
>>>>
>>>>
>>>> and experiment with them.
>>> Thank you, I have tried different approaches and they all seem to run
>>> more or less at the same speed (see below). I had to slightly modify
>>> your code from:
>>>
>>> table[i] = myrow
>>>
>>> to
>>>
>>> table[i] = [myrow]
>>>
>>> To avoid exceptions.
>>>
>>> In the newly attached file, I switched to blosc for compression (but
>>> with compression level 1) and run a few sensitivities. By calling the
>>> attached script as:
>>>
>>> python pytables_test.py NUM_SIM
>>>
>>> where "NUM_SIM" is an integer, I get the following timings and file 
>>> sizes:
>>>
>>> C:\MyProjects\Phaser\tests>python pytables_test.py 10
>>> Number of simulations   : 10
>>> H5 file creation time   : 0.879s
>>> Saving results for table: 6.413s
>>> H5 file size (MB)       : 193
>>>
>>>
>>> C:\MyProjects\Phaser\tests>python pytables_test.py 100
>>> Number of simulations   : 100
>>> H5 file creation time   : 4.155s
>>> Saving results for table: 86.326s
>>> H5 file size (MB)       : 1935
>>>
>>>
>>> I dont think I will try the 1,000 simulations case :-) . I believe I
>>> still don't understand what the best strategy would be for my problem.
>>> I basically need to save all the simulation results for all the 1,200
>>> "objects", each of which has a timeseries matrix of 600x7 size. In the
>>> GUI I have, these 1,200 "objects" are grouped into multiple
>>> categories, and multiple categories can reference the same "object",
>>> i.e.:
>>>
>>> Category_1: object_1, object_23, object_543, etc...
>>> Category_2: object_23, object_100, object_543, etc...
>>>
>>> So my idea was to save all the "objects" results to disk and, upon the
>>> user's choice, build the categories results "on the fly", i.e. by
>>> seeking the H5 file on disk for the "objects" belonging to that
>>> specific category and summing up all their results (over time, i.e.
>>> the 600 time-steps). Maybe I would be better off with a 4D array
>>> (NUM_OBJECTS, NUM_SIM, TSTEPS, 7) as a table, but then I will lose the
>>> ability to reference the "objects" by their names...
>>
>> You should keep trying experimenting with different approaches and 
>> discover the one that works for you the best.  Regarding using the 4D 
>> array as a table, I might be misunderstanding your problem, but you 
>> can still reference objects by name by using:
>>
>> row = table.where("name == %s" % my_name)
>> table[row.nrow] = ...
>
> Uh, I rather meant:
>
> row = table.readWhere("name == %s" % my_name)
> table[row.nrow] = ...
>
> but you probably got the idea already.
>

Ups, that does not work either.  It is probably something more like:

rowid = table.readWhereList("name == %s" % my_name)[0]
myrow = table[rowid]
table[rowid] = ...

(assuming that 'name' is a primary key here, i.e. values are not repeated).

-- 
Francesc Alted