Re: [Pytables-users] Table structure

[Thibault N. <tn...@fe...>]

On November 9, 2011 02:04:45 AM Francesc Alted wrote:
> 2011/11/8 Thibault North <tn...@fe...>:
>> [...]
> 
> For things like that, it is always recommended to get your big
> matrices out of Table objects.  In your case, you may want to put
> `steps` (and probably `output` too) in external EArray objects.  For
> example, if your table is going to be L rows large, then you can keep
> `steps` on a EArray with shape (L, K, N). With this, accessing your
> EArray data in typical table loops is easy:
> 
> for row in mytable:
>     niter = row['iteration']
>     myearray[row.nrow]
>     # operate with this
> 
> If (K,N) arrays are too long, EArray objects lets you fill (and
> retrieve) parts of it easily via indexing.  For example:
> 
>    myearray[my_l_idx, my_k_idx]
> 
> retrieves the row `my_k_idx` from `my_l_idx` element.  Similarly, you
> can update parts too:
> 
>    myearray[my_l_idx, my_k_idx] = np.arange(N, dtype=np.double)

Thanks for your detailled reply.

I am now using a EArray, as you suggested, with:
myearray = h5file.createEArray(h5file.root,  'steps', atom                                   
, (L, K, 0), "desc")
(Number of iteration N is unknown but usually small, L and K are known)

If I understand well, I need to append an initially empty (L,K) matrix for 
each iteration:
myearray.append(np.zeros((L,K))).reshape(L, K, 1))

and then fill it by iterating on the K dimension:
for line in mymatrix:
  myearray[line, :, iteration] = mydatacol

This actually works, but:
- the append() requires to load an array of size(L,K) in memory
- I get a warning (with L=2e4+1 and K=1e3) 
/usr/lib64/python2.7/site-packages/tables/leaf.py:416: PerformanceWarning: The 
Leaf ``/steps`` is exceeding the maximum recommended rowsize (104857600 
bytes);
be ready to see PyTables asking for *lots* of memory and possibly slow
I/O.  You may want to reduce the rowsize by trimming the value of
dimensions that are orthogonal (and preferably close) to the *main*
dimension of this leave.  Alternatively, in case you have specified a
very small/large chunksize, you may want to increase/decrease it.
  PerformanceWarning)
Is that just a matter of choosing the good position for the extendable 
dimension?

I quickly checked the docs, but couldn't find a way to "fill [...] parts of it 
[the array] easily via indexing". Append() seems to be required before...

Thanks again,
Thibault

> 
> Alternatively, you can also use a CArray here, with the difference
> that, with this, you should declare the final length of the container
> (while this is not necessary with EArray).  Read the documentation
> about EArray/CArray more carefully to explore their rich set of
> features.
> 
> Hope this helps,