Thread: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

Brought to you by: a_valentino, falted, ivilata, joshmoore

pytables-users

[Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

From: <phi...@ho...> - 2005-09-02 15:28:32

Hi Francesc and all others pytables users/developpers,

I have structured data: atmospheric pressure which depends from 
latitute, longitude and time.
The problem is i can't use pytables Table due to the application 
capabilities involved to manage the file.

The unique solution i have is to store each data in an array (pressure 
is a multidimensionnal array, time is an array, longitude is an array 
and latitude is an array).

Then i have to use a specific attributes to retrieve the right 
assiocation between parameters and pressure.

1. Is there a possibility to put a multidimensionnal attributes, which 
can consist in the list containing the path of the parameters 
([/longitude,/latitute,/time]) ?
So i know the first dimension of the pressure array is the longitude, 
the second the latitude, the third the time and the last one the values 
of the pressure.

I know there were a discussion and a patch about this but i don't know 
how to use it.

2. Would it be out of context to think about a standard way to link 
inside pytables, the dimension of a multidimensionnal array to the array 
defining the parameters values comparably to Netcdf ?

Thanks a lot for your answers,
regards,
Philippe Collet

Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

From: Francesc A. <fa...@ca...> - 2005-09-05 08:53:38

El dv 02 de 09 del 2005 a les 17:28 +0200, en/na
phi...@ho... va escriure:
> Hi Francesc and all others pytables users/developpers,
>=20
> I have structured data: atmospheric pressure which depends from=20
> latitute, longitude and time.
> The problem is i can't use pytables Table due to the application=20
> capabilities involved to manage the file.

Uh, you mean limitations specific of pytables or other kind of
limitations?

>=20
> The unique solution i have is to store each data in an array (pressure=20
> is a multidimensionnal array, time is an array, longitude is an array=20
> and latitude is an array).
>=20
> Then i have to use a specific attributes to retrieve the right=20
> assiocation between parameters and pressure.
>=20
> 1. Is there a possibility to put a multidimensionnal attributes, which=20
> can consist in the list containing the path of the parameters=20
> ([/longitude,/latitute,/time]) ?
> So i know the first dimension of the pressure array is the longitude,=20
> the second the latitude, the third the time and the last one the values=20
> of the pressure.

I don't know exactly what you are trying to do, but, in general, you can
save any kind of python object as an attribute. However, if this object
has not a direct mapping with a multimensional (or scalar) atomic (i.e.
byte, char, int or float) HDF5 type, then it's pickled.

> 2. Would it be out of context to think about a standard way to link=20
> inside pytables, the dimension of a multidimensionnal array to the array=20
> defining the parameters values comparably to Netcdf ?

Can you be a bit more explicit on what you want exactly?

Cheers,

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=E1rabos Coop. V.   Enjoy Data
 "-"

Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

From: <phi...@ho...> - 2005-09-06 14:08:49

Francesc Altet wrote:

>El dv 02 de 09 del 2005 a les 17:28 +0200, en/na
>phi...@ho... va escriure:
>  
>
>>Hi Francesc and all others pytables users/developpers,
>>
>>I have structured data: atmospheric pressure which depends from 
>>latitute, longitude and time.
>>The problem is i can't use pytables Table due to the application 
>>capabilities involved to manage the file.
>>    
>>
>
>Uh, you mean limitations specific of pytables or other kind of
>limitations?
>
>  
>
Sorry that doesn't seem clear.
No limitations of pytables.
It's much more that i'm not allowed to use table.
Arrays are more appreciated to achieve my goal .

>>The unique solution i have is to store each data in an array (pressure 
>>is a multidimensionnal array, time is an array, longitude is an array 
>>and latitude is an array).
>>
>>Then i have to use a specific attributes to retrieve the right 
>>assiocation between parameters and pressure.
>>
>>1. Is there a possibility to put a multidimensionnal attributes, which 
>>can consist in the list containing the path of the parameters 
>>([/longitude,/latitute,/time]) ?
>>So i know the first dimension of the pressure array is the longitude, 
>>the second the latitude, the third the time and the last one the values 
>>of the pressure.
>>    
>>
>
>I don't know exactly what you are trying to do, but, in general, you can
>save any kind of python object as an attribute. However, if this object
>has not a direct mapping with a multimensional (or scalar) atomic (i.e.
>byte, char, int or float) HDF5 type, then it's pickled.
>  
>
That could be a good starting point. I only need an ordered list 
containing path of each array of system parameters.
For example, the attribute could be 
['/parameters/longitude','/parameters/latitude','/parameters/time']

>  
>
>>2. Would it be out of context to think about a standard way to link 
>>inside pytables, the dimension of a multidimensionnal array to the array 
>>defining the parameters values comparably to Netcdf ?
>>    
>>
>
>Can you be a bit more explicit on what you want exactly?
>  
>
In fact, i'm joining Norbert Nemec in his post from 2005-01-24 11:51 to 
request a convention in pytables to store not only data but also a way 
to link the data.

Here is thye situation, better represented:

Longitude    [0.1     0.2     0.3     0.4]

Latitude [0.25  0.35  0.45 0.55]
                                                             latitude
                                                                ^

Pressure : [[1, 1.25 , 1.78 , 1.4],               | (0.25)
                [3.1 , 2.4 , 1.89 , 8.2],              | (0.35)
                [4.5 , 7.9 , 4.63 , 1],                 | (0.45)
                [1.2 , 3.7 , 4.26 , 6]]                 | (0.55)
                                                                 |
longitude   < ---------------------------------        
               (0.1)  (0.2)  (0.3) (0.4)                    

                                                               

If i am able to know that pressure atmospheric data are store depending 
first from the latitude then the longitude, i can easily get the 
pressure given a longitude and a latitude.

For example : i need the pressure for longitude 0.2 and latitude 0.45.

Then pressure is 7.9.

I need to build this thing with pytables.

Longitude is an array of shape (4L,), latitude is an array of shape 
(4L,) and pressure is an array of shape (4L,4L).

For the moment, with pytables, there is no system attribute(s) to know 
first dimension in pressure shape is latitude and second is longitude.
Can you give me a something (a specific attribute for example) available 
in the pytables API, to declare the order of use of each parameters in 
the pressure array.

Thanks a lot,
Regards,
Philippe Collet

>Cheers,
>
>  
>

Re: [Pytables-users] Suggestions for storing =?iso-8859-1?q?mutlidimensionnal=09data_in_several_pytables?= arrays

From: Francesc A. <fa...@ca...> - 2005-09-06 15:53:01

A Tuesday 06 September 2005 16:08, phi...@ho... va escriure:
> Sorry that doesn't seem clear.
> No limitations of pytables.
> It's much more that i'm not allowed to use table.
> Arrays are more appreciated to achieve my goal .

Ok. But it's a pity because using the filtering capabilities in Table
objects, like:

[r['pressure'] for r in table if r['latitude'] =3D=3D X and r['latitude'] =
=3D=3D Y]

is very powerful to do what you are willing to.

> That could be a good starting point. I only need an ordered list
> containing path of each array of system parameters.
> For example, the attribute could be
> ['/parameters/longitude','/parameters/latitude','/parameters/time']

Definitely, that could be a solution if you want to stick with arrays.

> In fact, i'm joining Norbert Nemec in his post from 2005-01-24 11:51 to
> request a convention in pytables to store not only data but also a way
> to link the data.
>
> Here is thye situation, better represented:
[snipped]

Ok. I understand better now. With that, what you want is something
similar to:

pressure.where(pressure.axis.longitude =3D=3D 0.2 and
               pressure.axis.latitude =3D=3D 0.45 )

isn't it?

Well, I must recognize that the idea is pretty nice. In addition, if
values in latitude and longitude have a direct mapping into indices
(as you seem to suggest), then a binary search on axis could be made
so that finding the indexes should be a very fast operation.

Well, I think I'll put this suggestion in our TODO file.

Thanks for your detailed explanation!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

From: <phi...@ho...> - 2005-09-07 15:14:09

Francesc Altet wrote:

>A Tuesday 06 September 2005 16:08, phi...@ho... va escriure:
>  
>
>>Sorry that doesn't seem clear.
>>No limitations of pytables.
>>It's much more that i'm not allowed to use table.
>>Arrays are more appreciated to achieve my goal .
>>    
>>
>
>Ok. But it's a pity because using the filtering capabilities in Table
>objects, like:
>
>[r['pressure'] for r in table if r['latitude'] == X and r['latitude'] == Y]
>
>is very powerful to do what you are willing to.
>  
>
Of course, using python list comprehension in tables is really 
powerfull. But i can't actually use tables. I have to use arrays.

>  
>
>>That could be a good starting point. I only need an ordered list
>>containing path of each array of system parameters.
>>For example, the attribute could be
>>['/parameters/longitude','/parameters/latitude','/parameters/time']
>>    
>>
>
>Definitely, that could be a solution if you want to stick with arrays.
>
>  
>
>>In fact, i'm joining Norbert Nemec in his post from 2005-01-24 11:51 to
>>request a convention in pytables to store not only data but also a way
>>to link the data.
>>
>>Here is thye situation, better represented:
>>    
>>
>[snipped]
>
>Ok. I understand better now. With that, what you want is something
>similar to:
>
>pressure.where(pressure.axis.longitude == 0.2 and
>               pressure.axis.latitude == 0.45 )
>
>isn't it?
>
>Well, I must recognize that the idea is pretty nice. In addition, if
>values in latitude and longitude have a direct mapping into indices
>(as you seem to suggest), then a binary search on axis could be made
>so that finding the indexes should be a very fast operation.
>
>Well, I think I'll put this suggestion in our TODO file.
>
>Thanks for your detailed explanation!
>  
>
That's exactly the idea behind all. And you go further with the 
suggestion about the direct mapping into indices. That could be very 
powerfull.
Thanks a lot for taking in account new improvements.
Of course, i can give you a hand if you need help.
Tell me when you start programming it, we can work together.
If you didn't have time, i will try soon to modify pytables code with 
your rigths.
Thanks a lot for your work.
Regards,
Philippe Collet