From: Francesc A. <fa...@py...> - 2010-05-20 18:29:09
|
Hi List, I'm happy to announce the first candidate release for PyTables 2.2 series. Among the most exciting improvements of this release is the support of threads in several parts of PyTables, namely Blosc and, optionally, Numexpr (which is out of the main distribution now and becomes a requisite). In particular, I'm quite happy of how performs the recent multi-threaded implementation that undergone Blosc in 0.9. It uses a pool of threads technique in order to reduce thread management to a bare minimum. When all the tests would be finished, I expect to release Blosc 1.0 very soon now (hopefully before PyTables 2.2 final). These additions will allow you to make full use of the raw speed of nowadays multi-core processors in the parts of the code that can use parallelism, and are only the beginning of a series of future multi-core improvements inside PyTables. Here it is the official announcement: =========================== Announcing PyTables 2.2rc1 =========================== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. This is the first release candidate for PyTables 2.2. On it, Numexpr is not included anymore and is now a requisite and the Blosc compressor has been updated to 0.9, which comes with integrated support for threads. Also, Cython is used per default now to build Pyrex extensions. Finally, a handful of bugs have been addressed and squashed. In case you want to know more in detail what has changed in this version, have a look at: http://www.pytables.org/moin/ReleaseNotes/Release_2.2rc1 You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://www.pytables.org/download/preliminary For an on-line version of the manual, visit: http://www.pytables.org/docs/manual-2.2rc1 Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- Francesc Alted |
From: Ivan V. i B. <iv...@se...> - 2010-06-07 11:40:09
|
Francesc Alted (2010-05-20 20:28:47 +0200) wrote: > [...] > I'm happy to announce the first candidate release for PyTables 2.2 series. > Among the most exciting improvements of this release is the support of threads > in several parts of PyTables, namely Blosc and, optionally, Numexpr (which is > out of the main distribution now and becomes a requisite). > > In particular, I'm quite happy of how performs the recent multi-threaded > implementation that undergone Blosc in 0.9. It uses a pool of threads > technique in order to reduce thread management to a bare minimum. When all > the tests would be finished, I expect to release Blosc 1.0 very soon now > (hopefully before PyTables 2.2 final). > [...] I've run the heavy test suite from repository version r4463 on an Intel Core 2 Duo running Debian unstable and all tests pass. Here is the sys-info:: PyTables version: 2.2rc2.dev HDF5 version: 1.8.4 NumPy version: 1.4.0 Numexpr version: 1.3.1 (not using Intel's VML/MKL) Zlib version: 1.2.3.4 (in Python interpreter) LZO version: 2.03 (Apr 30 2008) BZIP2 version: 1.0.5 (10-Dec-2007) Blosc version: 0.9.3 (2010-06-04) Python version: 2.5.4 (r254:67916, Feb 18 2009, 03:00:47) [GCC 4.3.3] Platform: linux2-x86_64 Byte-ordering: little The heavy suite tests from the 2.2rc1 tarball also pass in a G4 running Mac OS X Tiger, but I first had to apply r4453 to fix the issue with barriers during build time. Sys-info:: PyTables version: 2.2rc1 HDF5 version: 1.8.4-patch1 NumPy version: 1.3.0 Numexpr version: 1.3.1 (not using Intel's VML/MKL) Zlib version: 1.2.3 LZO version: 2.03 (Apr 30 2008) BZIP2 version: 1.0.5 (10-Dec-2007) Blosc version: 0.9.0 (2010-05-04) Python version: 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32) [GCC 4.0.1 (Apple Inc. build 5493)] Platform: darwin-Power Macintosh Byte-ordering: big Congrats for the new release! -- Ivan Vilata i Balaguer -- http://ivan.lovesgazpacho.net/ |
From: Francesc A. <fa...@py...> - 2010-06-07 12:49:50
Attachments:
blosc_bigendian.h5
blosc_bigendian.py
|
A Monday 07 June 2010 13:22:45 Ivan Vilata i Balaguer escrigué: > Francesc Alted (2010-05-20 20:28:47 +0200) wrote: > > [...] > > I'm happy to announce the first candidate release for PyTables 2.2 > > series. Among the most exciting improvements of this release is the > > support of threads in several parts of PyTables, namely Blosc and, > > optionally, Numexpr (which is out of the main distribution now and > > becomes a requisite). > > > > In particular, I'm quite happy of how performs the recent multi-threaded > > implementation that undergone Blosc in 0.9. It uses a pool of threads > > technique in order to reduce thread management to a bare minimum. When > > all the tests would be finished, I expect to release Blosc 1.0 very soon > > now (hopefully before PyTables 2.2 final). > > [...] > > I've run the heavy test suite from repository version r4463 on an Intel > Core 2 Duo running Debian unstable and all tests pass. Here is the > sys-info:: > > PyTables version: 2.2rc2.dev > HDF5 version: 1.8.4 > NumPy version: 1.4.0 > Numexpr version: 1.3.1 (not using Intel's VML/MKL) > Zlib version: 1.2.3.4 (in Python interpreter) > LZO version: 2.03 (Apr 30 2008) > BZIP2 version: 1.0.5 (10-Dec-2007) > Blosc version: 0.9.3 (2010-06-04) > Python version: 2.5.4 (r254:67916, Feb 18 2009, 03:00:47) [GCC 4.3.3] > Platform: linux2-x86_64 > Byte-ordering: little > > The heavy suite tests from the 2.2rc1 tarball also pass in a G4 running Mac > OS X Tiger, but I first had to apply r4453 to fix the issue with barriers > during build time. Sys-info:: > > PyTables version: 2.2rc1 > HDF5 version: 1.8.4-patch1 > NumPy version: 1.3.0 > Numexpr version: 1.3.1 (not using Intel's VML/MKL) > Zlib version: 1.2.3 > LZO version: 2.03 (Apr 30 2008) > BZIP2 version: 1.0.5 (10-Dec-2007) > Blosc version: 0.9.0 (2010-05-04) > Python version: 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32) [GCC 4.0.1 > (Apple Inc. build 5493)] Platform: darwin-Power Macintosh > Byte-ordering: big > > Congrats for the new release! Great Ivan! :-) Provided that you have a big-endian machine, I'd ask you (or anybody else having such a platform) to help me testing the new capability in Blosc 0.9.4 for being cross-compatible among little/big endian machines. Could you please run the attached script (blosc_bigendian.py) and send the resulting file back to me? I plan to add this file in the test suite. Also, please send me the output of ``ptdump -vd`` against my attached little-endian file (blosc_bigendian.h5). You will need to update your PyTables trunk local repository in your big-endian platform before doing this, of course. Many thanks! -- Francesc Alted |
From: Ivan V. i B. <iv...@se...> - 2010-06-08 17:29:22
Attachments:
blosc_bigendian.dump
blosc_bigendian.h5
|
Francesc Alted (2010-06-07 14:49:34 +0200) wrote: > Could you please run the attached script (blosc_bigendian.py) and send the > resulting file back to me? I plan to add this file in the test suite. Also, > please send me the output of ``ptdump -vd`` against my attached little-endian > file (blosc_bigendian.h5). You will need to update your PyTables trunk local > repository in your big-endian platform before doing this, of course. I'm attaching the result of dumping your attached HDF5 file and the new HDF5 file of the same name which is created by the script. I've used PyTables SVN trunk r4464 under the G4. -- Ivan Vilata i Balaguer -- http://ivan.lovesgazpacho.net/ |
From: Francesc A. <fa...@py...> - 2010-06-09 15:06:12
|
A Tuesday 08 June 2010 19:29:16 Ivan Vilata i Balaguer escrigué: > Francesc Alted (2010-06-07 14:49:34 +0200) wrote: > > Could you please run the attached script (blosc_bigendian.py) and send > > the resulting file back to me? I plan to add this file in the test > > suite. Also, please send me the output of ``ptdump -vd`` against my > > attached little-endian file (blosc_bigendian.h5). You will need to > > update your PyTables trunk local repository in your big-endian platform > > before doing this, of course. > > I'm attaching the result of dumping your attached HDF5 file and the new > HDF5 file of the same name which is created by the script. I've used > PyTables SVN trunk r4464 under the G4. Great, it works as intended. Thanks! -- Francesc Alted |