pytables-users

[Pytables-users] subarray extraction and update

[Bernard K. <ber...@be...>]

Dear all,

I am using pytables mostly to store data in the form of simple 
two-dimensional arrays. For my calculations I often need to extract and 
update submatrices of these arrays. Unfortunately the row and column 
indexes I am using can not be described as slices because those can be 
quite random.

Can anyone teach me a "good" (meaning memory efficient, fast and 
elegant) way to code submatrix extraction and update ?
The ideal for me would be a code close to this one :

x=array([1,2,5,8])   # row index
y=array([0,3,7])   # column index
a=data[x,y]   # returns an array of size [4,3], 'data' is a pytables 2-d 
array
data[x,y]=0   # set the submatrix defined by x rows and y columns to 0
data[x,y]=rand([4,3])  # update of a submatrix
data[x,y]=data2[x,y]   # copy of a submatrix from one array into another

Sincerely,

Bernard KAPLAN

Re: [Pytables-users] subarray extraction and update

[Antonio V. <val...@co...>]

Alle 11:36, luned=EC 25 luglio 2005, Bernard KAPLAN ha scritto:
> Dear all,

hi Bernard

[...]

> x=3Darray([1,2,5,8])   # row index
> y=3Darray([0,3,7])   # column index
> a=3Ddata[x,y]   # returns an array of size [4,3], 'data' is a pytables =
2-d
> array
> data[x,y]=3D0   # set the submatrix defined by x rows and y columns to =
0
> data[x,y]=3Drand([4,3])  # update of a submatrix
> data[x,y]=3Ddata2[x,y]   # copy of a submatrix from one array into anot=
her

I think that what you are asking for can't be done in an "elegant" way.
HDF5 supports hyperslabs selection that is powerfull and elegant but it=20
requires reqular spacing in the selection, see=20

http://hdf.ncsa.uiuc.edu/HDF5/doc/RM_H5S.html#Dataspace-SelectHyperslab

In order to use hyperslabs you have to be able to express the sub-matrix=20
selection in terms of "start, stride, count, and block".
It seems to me that it is not your case.

> Sincerely,
>
> Bernard KAPLAN
>
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_id=3D7477&alloc_id=3D16492&op=3Dcl=
ick
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

ciao

--=20
Antonio Valentino

Re: [Pytables-users] subarray extraction and update

[Bernard K. <ber...@be...>]

Dear Antonio,

In hdf5 you can define progressively a selected region (adding elements=20
one by one) by use of either "H5Sselect=5Felements" with H5S=5FSELECT=5FAPP=
END=20
or "H5Sselect=5Fhyperslab" with H5S=5FSELECT=5FOR. Then you can read or wri=
te=20
the selected elements at once with "H5Dread" or "H5Dwrite".

Is there a function in pytables that use this functionality ? How would=20
you do to extract a submatrix in pytables ?

Bernard||

Antonio Valentino wrote:

>Alle 11:36, luned=EC 25 luglio 2005, Bernard KAPLAN ha scritto:
> =20
>
>>Dear all,
>>   =20
>>
>
>hi Bernard
>
>[...]
>
> =20
>
>>x=3Darray([1,2,5,8])   # row index
>>y=3Darray([0,3,7])   # column index
>>a=3Ddata[x,y]   # returns an array of size [4,3], 'data' is a pytables 2-d
>>array
>>data[x,y]=3D0   # set the submatrix defined by x rows and y columns to 0
>>data[x,y]=3Drand([4,3])  # update of a submatrix
>>data[x,y]=3Ddata2[x,y]   # copy of a submatrix from one array into another
>>   =20
>>
>
>I think that what you are asking for can't be done in an "elegant" way.
>HDF5 supports hyperslabs selection that is powerfull and elegant but it=20
>requires reqular spacing in the selection, see=20
>
>http://hdf.ncsa.uiuc.edu/HDF5/doc/RM=5FH5S.html#Dataspace-SelectHyperslab
>
>In order to use hyperslabs you have to be able to express the sub-matrix=20
>selection in terms of "start, stride, count, and block".
>It seems to me that it is not your case.
>
> =20
>
>>Sincerely,
>>
>>Bernard KAPLAN
>>
>>
>>
>>-------------------------------------------------------
>>SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
>>from IBM. Find simple to follow Roadmaps, straightforward articles,
>>informative Webcasts and more! Get everything you need to get up to
>>speed, fast. http://ads.osdn.com/?ad=5Fid=3D7477&alloc=5Fid=3D16492&op=3D=
click
>>=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
>>Pytables-users mailing list
>>Pyt...@li...
>>https://lists.sourceforge.net/lists/listinfo/pytables-users
>>   =20
>>
>
>ciao
>
> =20
>

Re: [Pytables-users] subarray extraction and update

[Antonio V. <val...@co...>]

Alle 16:44, luned=EC 25 luglio 2005, Bernard KAPLAN ha scritto:
> Dear Antonio,
>
> In hdf5 you can define progressively a selected region (adding elements
> one by one) by use of either "H5Sselect_elements" with H5S_SELECT_APPEN=
D
> or "H5Sselect_hyperslab" with H5S_SELECT_OR. Then you can read or write
> the selected elements at once with "H5Dread" or "H5Dwrite".

Yes, you are right. The selection operation is not as immediate as in you=
r=20
example, but afrer selection you can read all data at once.

> Is there a function in pytables that use this functionality ? How would
> you do to extract a submatrix in pytables ?

I think that at the moment this functionality is not available. Surely=20
Francesk or Ivan can give you more explainations.
Usually I get an entire block of data from arrays=20

data =3D h5f.myarray[0:100, 0:100]

I never encountered a situation like yours, sorry :((

> Bernard||
>
> Antonio Valentino wrote:
> >Alle 11:36, luned=EC 25 luglio 2005, Bernard KAPLAN ha scritto:
> >>Dear all,
> >
> >hi Bernard
> >
> >[...]
> >
> >>x=3Darray([1,2,5,8])   # row index
> >>y=3Darray([0,3,7])   # column index
> >>a=3Ddata[x,y]   # returns an array of size [4,3], 'data' is a pytable=
s 2-d
> >>array
> >>data[x,y]=3D0   # set the submatrix defined by x rows and y columns t=
o 0
> >>data[x,y]=3Drand([4,3])  # update of a submatrix
> >>data[x,y]=3Ddata2[x,y]   # copy of a submatrix from one array into an=
other

[...]

> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id=16492&op,ick
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

--=20
Antonio Valentino

Re: [Pytables-users] subarray extraction and update

[Francesc A. <fa...@ca...>]

A Monday 25 July 2005 16:44, Bernard KAPLAN va escriure:
> In hdf5 you can define progressively a selected region (adding elements
> one by one) by use of either "H5Sselect_elements" with H5S_SELECT_APPEND
> or "H5Sselect_hyperslab" with H5S_SELECT_OR. Then you can read or write
> the selected elements at once with "H5Dread" or "H5Dwrite".

That's correct. In fact, H5Sselect_elements is already used by
Table.readCoordinates. Applying it to support indexing in your
suggested way was already proposed by Pete in this list a few days
ago, but just for Table objects but expanding its use to *Array
objects would be nice (although not high priority for us right now).

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] subarray extraction and update

[travlr <vel...@gm...>]

On 7/26/05, Francesc Altet <fa...@ca...> wrote:
>=20
> A Monday 25 July 2005 16:44, Bernard KAPLAN va escriure:
> > In hdf5 you can define progressively a selected region (adding elements
> > one by one) by use of either "H5Sselect_elements" with H5S_SELECT_APPEN=
D
> > or "H5Sselect_hyperslab" with H5S_SELECT_OR. Then you can read or write
> > the selected elements at once with "H5Dread" or "H5Dwrite".
>=20
> That's correct. In fact, H5Sselect_elements is already used by
> Table.readCoordinates. Applying it to support indexing in your
> suggested way was already proposed by Pete in this list a few days
> ago, but just for Table objects but expanding its use to *Array
> objects would be nice (although not high priority for us right now).
>=20
> Cheers,



Although my coding experience is still fairly green, and time is pretty=20
tight, It'd be a pleasure to contribute to pytables. I'll try to see if I=
=20
can muster up the know how to implement this for us. Much is appreciated of=
=20
the Carabos crew.

regards,
Pete


--
> >0,0< Francesc Altet http://www.carabos.com/
> V V C=E1rabos Coop. V. Enjoy Data
> "-"
>=20
>=20
>=20
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id=16492&opclick
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

Re: [Pytables-users] subarray extraction and update

[Francesc A. <fa...@ca...>]

A Wednesday 27 July 2005 13:51, travlr va escriure:
> Although my coding experience is still fairly green, and time is pretty
> tight, It'd be a pleasure to contribute to pytables. I'll try to see if I
> can muster up the know how to implement this for us. Much is appreciated =
of
> the Carabos crew.

Yes, no problem. My personal advice is that you look at how
readCoordinates makes use of the H5Sselect_elements, and apply the
same to arrays. Read the excellent HDF5 docs available in:

http://hdf.ncsa.uiuc.edu/HDF5/doc/

Do not hesitate asking on this list if you get stuck. The C=E1rabos crew
will in vacancies for most of the month of August, but some of us will
check the e-mail from time to time. Also, I know that there are quite
a few other people in the list having the necessary skills to answer
to your questions.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"