Menu
▾
▴
pytables-users
|
From: Francesc A. <fa...@ca...> - 2005-11-09 10:48:29
|
Hi Remi, A Dimecres 09 Novembre 2005 10:59, v=E0reu escriure: > Hi, > I'm currently planning to add Dimension Scales to PyTables. Actually > Cyril Giraudon (do you remember him ?) told me about this idea, which > I think is a great one. Given that I've more time to spend working on > it than Cyril, I've decided to make it happen. Excellent news :-) > I first plan to limit myself to one dimensional Array objects, then > I'll extend it. I've had a look at PyTables sources, and to my mind > adding these Dimension Scale shouldn't be very big a problem. Sounds good > So my question is the following : how do you build "hdf5Extension.so" > ? Do you use a makefile, or is this build performed by the "setup.py"? It's very easy. Just issue a: python setup.py build_ext --inplace and distutils will be in charge of computing dependencies an recompile everything that is needed. Have in mind that you need Pyrex installed before doing this. The --inplace flag means that the extensions will be put in the tables/ directory so that you can do your tests without a need to install the complete beast (remember to add your working tables/ directory to your PYTHONPATH environment variable first). Good luck! =2D-=20 >0,0< Francesc Altet =A0 =A0 http://www.carabos.com/ V V C=E1rabos Coop. V. =A0=A0Enjoy Data "-" |
|
From: Francesc A. <fa...@ca...> - 2005-11-09 10:52:10
|
Ops, I forgot to say that a good starting point for modifying the sources is to fetch the 1.2rc2 release in: http://www.carabos.com/downloads/pytables/preliminary/pytables-1.2-rc2.tar.= gz which will hopefully be the definitive 1.2 version, bar some arrangements on the array protocol for doing numarray<-->Numeric conversions. Cheers, =2D-=20 >0,0< Francesc Altet =A0 =A0 http://www.carabos.com/ V V C=E1rabos Coop. V. =A0=A0Enjoy Data "-" |
|
From: Francesc A. <fa...@ca...> - 2005-11-10 17:35:48
|
R=E9mi, A Dijous 10 Novembre 2005 17:37, R=E9mi Sassolas va escriure: > The generation of the C file from the pyrex file works fine, and no > errors are reported during the following compliation process (just > some Warning which are not related to my additions). > As for the linking process, no errors reported too. > But when I try to import the modified tables module, python complains > about an undefined symbol : H5DSset_scale, which as you may have > guessed is the name of the first function from the Dimension Scale C > Api I would like to add to PyTables. I've checked your version of the extension and it compiles fine to me. Are you directing the distutils to use the HDF5 1.7.x headers?. I normally use something like: python setup.py build_ext --inplace --hdf5=3D/the_path_to_your_hdf5-1.7.x/h= df5 and that's all. BTW, try to send attachments below 40 KB to the list, as it is a netiquette rule. =2D-=20 >0,0< Francesc Altet =A0 =A0 http://www.carabos.com/ V V C=E1rabos Coop. V. =A0=A0Enjoy Data "-" |
|
From: <rem...@gm...> - 2005-11-14 09:10:41
|
Hi,
2005/11/10, Francesc Altet <fa...@ca...>:
> R=E9mi,
>
> I've checked your version of the extension and it compiles fine to me.
> Are you directing the distutils to use the HDF5 1.7.x headers?. I
> normally use something like:
>
> python setup.py build_ext --inplace --hdf5=3D/the_path_to_your_hdf5-1.7.x=
/hdf5
>
> and that's all.
I may not have been clear. Nothing goes - at least seems to go - wrong
when I try to compile the modified extension (using the "setup.py"
just as you do).
But when I try to import PyTables (or even directly the
"hdf5Extension.so"), Python complains about an undefined symbol
("H5DSset_scale", which is the name of the first function I've tried
to add from the Dimension Scale APi).
After having compiled the modified extension, have you tried to import
PyTables, or the "hdf5Extension.so" ?
>
> BTW, try to send attachments below 40 KB to the list, as it is a
> netiquette rule.
>
Oups I'm sorry about that, I didn't pay enough attention.
|
|
From: Francesc A. <fa...@ca...> - 2005-11-14 09:38:58
|
El dl 14 de 11 del 2005 a les 10:10 +0100, en/na R=E9mi Sassolas va
escriure:
> I may not have been clear. Nothing goes - at least seems to go - wrong
> when I try to compile the modified extension (using the "setup.py"
> just as you do).
> But when I try to import PyTables (or even directly the
> "hdf5Extension.so"), Python complains about an undefined symbol
> ("H5DSset_scale", which is the name of the first function I've tried
> to add from the Dimension Scale APi).
> After having compiled the modified extension, have you tried to import
> PyTables, or the "hdf5Extension.so" ?
Oh, I see. I think this can be due to the fact that you forgot to point
your LD_LIBRARY_PATH to your HDF5 1.7.x installation directory (most
probably in /the_path_to_your_hdf5-1.7.x/hdf5/lib).
Hope that helps,
--=20
>0,0< Francesc Altet http://www.carabos.com/
V V C=E1rabos Coop. V. Enjoy Data
"-"
|
|
From: <rem...@gm...> - 2005-11-14 10:13:12
Attachments:
setup.py.tar.gz
|
2005/11/14, Francesc Altet <fa...@ca...>: > Oh, I see. I think this can be due to the fact that you forgot to point > your LD_LIBRARY_PATH to your HDF5 1.7.x installation directory (most > probably in /the_path_to_your_hdf5-1.7.x/hdf5/lib). > > Hope that helps, Hi, I've found how to fix my problem. In fact it was rather simple (I must admit that I'm a little ashamed...). When I look more closely at what the setup.py does, I've realized that during the linking process, there was only one library from Hdf5 which was included : "libhdf5". But the functions in the Dimension Scale API don't belong to this library, but to "libhdf5_hl". So I've modified the setup.py, to add this missing library. And then I don't get any error message from Python. For the moment I've only modified the code for the linux part, but it should be easy to extend it to the other os. Now I can start to design the Python class for the Dimension Scale which can use these methods. I join to this mail the modified setup.py -- R=E9mi |
|
From: Francesc A. <fa...@ca...> - 2005-11-14 11:05:59
|
El dl 14 de 11 del 2005 a les 11:13 +0100, en/na R=E9mi Sassolas va escriure: > I've found how to fix my problem. In fact it was rather simple (I must > admit that I'm a little ashamed...). When I look more closely at what > the setup.py does, I've realized that during the linking process, > there was only one library from Hdf5 which was included : "libhdf5". > But the functions in the Dimension Scale API don't belong to this > library, but to "libhdf5_hl". So I've modified the setup.py, to add > this missing library. And then I don't get any error message from > Python. Don't be ashamed. You have discovered now how to add more library dependencies in PyTables ;-) > For the moment I've only modified the code for the linux part, but it > should be easy to extend it to the other os. Yeah, I think so. Cheers, --=20 >0,0< Francesc Altet http://www.carabos.com/ V V C=E1rabos Coop. V. Enjoy Data "-" |
|
From: Francesc A. <fa...@ca...> - 2005-11-29 20:12:53
|
Hi R=E9mi, I'm out for a workshop on HDF in San Francisco. When I go back to Spain (next week), I'll check your patches more in detail and will give you my feedback shortly after. Maybe Ivan would like to have a look at it meanwhile. Please, remember to attach just your patches, not the entire code. Also, remember that you should not send attachements bigger than 40 KB to this list. If you want to make your patches publicly available, you can submit it to SF: http://sourceforge.net/tracker/?group_id=3D63486&atid=3D504146 Thank you very much for your contribution! =46rancesc A Dilluns 28 Novembre 2005 18:40, rem...@gm... va escriure: > > Hi, > > It's been quite a long time since I sent you last news about Dimension > > Scales. Don't worry, I have not stayed iddle all this time long. > > I join to this mail an archive of what I've done so far. I've only put = in > > this archive (as you may guess) the files from PyTables that I have > > modified. > > I shall precise that the files I've modified are the ones from PyTables > > v1.2 (I've started to work with a release candidate, and then I shifted= - > > without any problems - to the lattest release). > > If you encounter any problem in trying to make Dimension Scales work, > > feel free to ask me. > > > > Here is a link for the specification for Dimension Scales, from the HDF5 > > team. It 's not fully up to date, but to my mind it's interesting (it c= an > > help you understand the "philosophy" of Hdf5 Dimension Scales). > > -> http://hdf.ncsa.uiuc.edu/RFC/H5DimScales/ > > Here is a summary of what I've done : > > - I've added the functions of the DS Api into the "hdf5Extension.pyx", > > and then created new methods for the class hdf5Extension.Array which use > > these functions. That was rather easy. > > > > - Then I've added new methods for the class Array of tables.Array, which > > call these (inherited) methods and add exception handling stuff. > > Then I realized that an exception would always be raised when I try to > > open a hdf5 file which contains a Dimension Scale. > > Actually the hdf5 class name didn't correspond to the '_c_classId' of t= he > > PyTables Python class (it's badly explained but I hope you'll get the > > idea). > > > > - So I created a new class on the Python side, with a new '_c_classId'= : > > 'DIMENSION_SCALE'. I had to put this class in the same file as > > tables.Array (ie "Array.py") to avoid a loop in the Python imports : > > DimensionScale should inherit from Array, and so should import it, but > > Array should also import DimensionScale. > > There I use a little trick. I needed some kind of a "cast" to turn an > > existing Array into a DimensionScale. The trick merely consists in > > modifying the attribute __class__ . Simple and pretty efficient. It > > doesn't sound very clean, but I haven't found a way to work around my > > problem differently. > > > > - Then I started tackling the special attributes for DS, which turn out > > to be the biggest difficulty in my work. > > I first needed to understand the code related to these attribute in hdf5 > > code, and then add it into hdf5Extension.pyx. I was not sure whether I > > should declare them as system attributes, so in the end I declared them > > as public ones (but I protect them). > > > > Currently these attributes are read from hdf5 whenever one access them. > > I'll try to improve this in the coming days. > > > > Two of these attributes are "reference list". The "DIMENSION_LIST" is an > > attribute for an Array, it stores references on the Dimension Scales th= at > > are attached to it. Conversely the "REFERENCE_LIST" is an attribute for= a > > Dimension Scale, it stores references on the Arrays it is attached to. > > These references consist of four digit numbers (and an other number to > > specify the index of the dimension concerned). The problem is that I've > > not managed in fully retrieving the target of a reference from its > > reference... > > > > I succeeded in retrieving the number of dimension, the number of points, > > the type, etc...But the name of the target seems to be lost. I've asked > > one of the hdf5 team about this point, and he didn't know how I can get > > it back. Actually from the reference one can retrieve an ID, which is n= ot > > the ID of the target but a new ID pointing to it. There are some infos - > > including the name - that can't be obtained from this new ID... > > > > But for example, let's assume "(ref, idx)" is a tuple from the > > "REFERENCE_LIST" of a DImension Scale "ds" ("ds" is attached to the > > dimension "idx" of the Array the reference of which is "ref"). > > From "ref" one can retrieve a new ID for this Array (using > > 'H5Rdereference'). Then from this new ID one can detach "ds" from the > > Array (but one can't get the name of this Array !). I've taken advantage > > of this to create two new methods which allow a Dimension Scale to deta= ch > > itself from all the Arrays it was attached to and to allow an Array to > > detach itself from all the Dimension that were attached to it . > > > > In the end, it's maybe not that usefull for an user to see these > > attributes (look at them and you'll understand ;-)). But they were very > > usefull for me to test my work, and I spent so much time on them...but = if > > you want me to hide them, it's not a problem. > > > > - If you don't understand all this stuff about attributes, play a little > > with DS and you'll understand. > > I've made two small test files (one for the DS and the other focused on > > the attributes of DS). > > I was too lazzy to make test files which look like the ones from > > PyTables, but I could improve it. > > > > - Finally, I've thought a little about how I could extend DS to CArrray= s, > > EArrays, etc. It doesn't sound very difficult, I've only done it for > > CArray - but I haven't tested it so far. Actually I've done the easiest > > part. Indeed I don't know how to deal with the enlargeable dimension. > > I've some ideas, but I would be interested in knowing yours (and this > > mail seems long enough to me :-)). > > > > If you have any question about this mail or DS or whatever, feel free = to > > ask me ! > > Cheers > > -- > > Remi =2D-=20 >0,0< Francesc Altet =A0 =A0 http://www.carabos.com/ V V C=E1rabos Coop. V. =A0=A0Enjoy Data "-" |
|
From: dragan s. <sav...@ya...> - 2005-12-01 14:54:06
|
Hi! Francesc: You posted a mail few months ago with this content: I'd liked to include the patch sent by travlr, to be able to use indexing to get and set values in *Array, Table and Column objects (i.e. "indexing a la numarray"); but provided he sent a patch just for getting values in Table and Column objects and that we are rather busy right now, we've decided to not incorporate this functionality until a next version, when all the objects would receive complete support for both getting and setting values "a la numarray". Is it possible to get this patch/code? I am very interested in this feature! Thanks! Regards, Dragan. __________________________________ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com |
|
From: Francesc A. <fa...@ca...> - 2005-12-07 11:13:18
|
Hi Remi, Let me some days to digest your contributions and send back my feedback to you. An issue that concerns me a little bit is whether we should include your patches in the next stable version of PyTables or not. The problem here is that dimension scales will be implemented in forthcoming HDF5 1.8, but not in HDF5 1.6.x, which is the current production version of HDF5. I think that the best would be to create a branch for dimension scales in the PyTables SVN repository, and continue the work there until HDF5 1.8 would hit the street. However, there is currently not provision of offering snapshots for SVN branches :-(. Anyway, this remember us that we should start implementing a public interface for the complete PyTables repository. We probably will choose Trac (http://projects.edgewall.com/trac/) for this, but feedback is welcome. I'll tell you more on this soon. Thanks! A Dimarts 06 Desembre 2005 17:34, rem...@gm... va escriure: > Hi, > Please accept my apologies for spamming so heavily this list. > I've made small improvements to my work : > - added a new class in hdf5Extension to better distinguish between 'simpl= e' > Arrays and Dimension Scales > - comments updated > - removed the method get_name(), given that this method merely used to get > the attribute 'NAME'. I can have it back if you wish. > Similarly I'm not convinced of the usefullness of the method get_label(). > - some small bugs fixed > - added a new test file for the new attributes and the 'cleaning' functio= ns > > I join to this mail the patch to be applied on Pytables ver 1.2 > I was too lazzy to generate the patches to be applied on the already > patched sources, but feel free to ask if you need them. > Cheers > -- > Remi =2D-=20 >0,0< Francesc Altet =A0 =A0 http://www.carabos.com/ V V C=E1rabos Coop. V. =A0=A0Enjoy Data "-" |
|
From: <rem...@gm...> - 2005-12-09 09:59:30
|
Hi Francesc, First don't worry since I sent my last patch I've not made new modifications. But there's something I've done that I feel uncomfortable with (hence this mail). Indeed I'm beginning to wonder whether adding new attributes (the ones storing pathnames instead of 'references') was so good an idea. Thanks to these attributes, I could almost get rid of the functions from th= e HDF5 Api for Dimension Scales. But my goal was not to redo Dimension Scales for PyTables, but rather to integrate the feature from Hdf5. Besides redoin= g a Hdf5 feature would - to my mind - be contrary to the PyTables philosophy= . If I could get the pathname of a dataset from its 'reference', I wouldn't have to add these attributes. And my work could be greately simplified. I've asked one of the Hdf5 team about this point, but according to him it's currently not possible. But maybe Hdf5 could be slightly modified, or maybe you know an other way t= o work around my problem (given that you certainly know much more about Hdf5 than I do). By the way, I fully agree with you about my work not being integrated in th= e next stable release of PyTables (unless Hdf5 1.8 is out). Cheers, -- Remi |
|
From: Francesc A. <fa...@ca...> - 2005-12-09 17:21:19
|
A Divendres 09 Desembre 2005 10:52, rem...@gm... va escriure: > But there's something I've done that I feel uncomfortable with (hence this > mail). Indeed I'm beginning to wonder whether adding new attributes (the > ones storing pathnames instead of 'references') was so good an idea. > Thanks to these attributes, I could almost get rid of the functions from > the HDF5 Api for Dimension Scales. But my goal was not to redo Dimension > Scales for PyTables, but rather to integrate the feature from Hdf5. Besid= es > redoing a Hdf5 feature would - to my mind - be contrary to the PyTables=20 > philosophy. If I could get the pathname of a dataset from its 'reference', > I wouldn't have to add these attributes. And my work could be greately > simplified. I've asked one of the Hdf5 team about this point, but accordi= ng > to him it's currently not possible. > But maybe Hdf5 could be slightly modified, or maybe you know an other way > to work around my problem (given that you certainly know much more about > Hdf5 than I do). Well, I am not very used to references in HDF5 (in fact, this is my first contact with them), but it is indeed unfortunate that you cannot derive the pathname of a dataset by dereferencing a HDF5 reference. I would suggest you to continue maintaining the REF_LIST and DIM_LIST attributes in order to keep track of the different dimension scales associations. Of course, this represents a deviation from the 'standard', but as it is an addition and not a modification, I think it is not grave at all (in fact, PyTables has many other additional attributes than the 'standard' in HDF5_HL objects). If in the future we can figure out how to derive pathnames from dereferences, then we can consider getting rid of the REF_LIST and DIM_LIST attributes. Meanwhile, I do hope that it won't be too difficult maintaining those attributes in your implementation of dimension scales. Cheers, =2D-=20 >0,0< Francesc Altet =A0 =A0 http://www.carabos.com/ V V C=E1rabos Coop. V. =A0=A0Enjoy Data "-" |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X