pytables-users — PyTables users discussion list

Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Christoph G. <cg...@uc...>]

On 8/27/2012 10:58 AM, Stuart Mentzer wrote:
> On 8/27/2012 1:22 PM, Christoph Gohlke wrote:
>> On 8/27/2012 9:42 AM, Antonio Valentino wrote:
>>> Hi Stuart,
>>>
>>> Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
>>>> Hello,
>>>>
>>>> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:
>>>>
>>>>          if not ctypes.util.find_library('hdf5dll.dll'):
>>>>              raise ImportError('Could not load "hdf5dll.dll", please ensure' +
>>>>                      ' that it can be found in the system path')
>>>>
>>>> PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.
>>>>
>>>> That code was added in revision e9f6919.
>>>>
>>> It is mainly a sanity check added under request of one of our users:
>>> https://github.com/PyTables/PyTables/pull/146
>>>
>>>
>>>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.
>>>>
>>>> Should I file a bug report? Any easy work-around?
>>>>
>>>> Thanks,
>>>> Stuart
>>>>
>>> Yes please file a pull request with your patch.
>>> It would be nice to preserve the sanity check in standard case so,
>>> maybe, a good solution could be adding some check on sys.frozen or
>>> something like that.
>>>
>>> Thank you
>>>
>> Hello,
>>
>> As a workaround for frozen distributions, try to add the sys.executable
>> directory to os.environ['PATH'] before importing tables.
>>
>> Ctypes only tries to find a library in the os.environ['PATH']
>> directories, not the current directory or the sys.executable directory
>> as one could expect.
>> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48
>>
>> As a workaround, for distributions that place the HDF5 and other DLLs in
>> the tables package directory, tables.__init__.py adds the tables package
>> directory to os.environ['PATH']. This also makes sure that the DLLs are
>> found when loading the hdf5Extension.pyd and other C extension modules
>> (another common problem). The use of __file__ to get the tables
>> directory should better be wrapped in a try..except statement.
>> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
>>
>> Christoph
> Hi Christoph,
>
> Thanks for the info/suggestions. It might be nice to add your comments to the
> Issue #177 I created.
>
> I was aware that altering the PATH is a work-around. Patching PyTables is cleaner
> and seems like the proper fix, and I think we agree that there is a problem here
> that should be addressed. Maybe the PyTables test suite should even include a
> frozen application test.
>
> Thanks,
> Stuart
>

Hi Stuart,

I'll try to work on a patch tonight. It's probably better to use Ctypes 
LoadLibrary instead of find_library because that makes sure all HDF5 
dependencies are found, it (supposedly) searches the Windows DLL search 
path not just os.environ['PATH'], and (supposedly) takes into account 
libraries already loaded into the process.

Christoph

Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Stuart M. <Stu...@ob...>]

On 8/27/2012 1:52 PM, Antonio Valentino wrote:
> Hi Christoph,
>
> thank you very much for your hints.
>
> Il 27/08/2012 19:22, Christoph Gohlke ha scritto:
>> On 8/27/2012 9:42 AM, Antonio Valentino wrote:
>>> Hi Stuart,
>>>
>>> Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
>>>> Hello,
>>>>
>>>> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:
>>>>
>>>>          if not ctypes.util.find_library('hdf5dll.dll'):
>>>>              raise ImportError('Could not load "hdf5dll.dll", please ensure' +
>>>>                      ' that it can be found in the system path')
>>>>
>>>> PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.
>>>>
>>>> That code was added in revision e9f6919.
>>>>
>>> It is mainly a sanity check added under request of one of our users:
>>> https://github.com/PyTables/PyTables/pull/146
>>>
>>>
>>>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.
>>>>
>>>> Should I file a bug report? Any easy work-around?
>>>>
>>>> Thanks,
>>>> Stuart
>>>>
>>> Yes please file a pull request with your patch.
>>> It would be nice to preserve the sanity check in standard case so,
>>> maybe, a good solution could be adding some check on sys.frozen or
>>> something like that.
>>>
>>> Thank you
>>>
>> Hello,
>>
>> As a workaround for frozen distributions, try to add the sys.executable
>> directory to os.environ['PATH'] before importing tables.
>>
> Christoph, I suppose it can also be done at this point too:
> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
> Isn't it?
>
> Please Stuart, can you try this fix as well.
Can't try it right now but I can't see why adding sys.excutable to the PATH at that point
wouldn't "work" as an alternative to skipping the sanity check when frozen. My broader
concern is whether fiddling with the PATH is the best or really the correct fix. If you
need to exactly replicate Windows DLL search rules (which, just to make life fun, vary
depending on the Windows version and the state of the SafeDllSearchMode*) then you should
do that rather than hacking the PATH. For example, what if there is another hdf5dll.dll
already on the PATH? In that case you'd better put sys.executable in the front of the PATH,
but that might break something else. Maybe 99.9% of the time the PATH hack works but
it can also fail.

I think the only proper fix is to make find_library follow the Windows rules exactly.
Short of that, it is not safe to use it when the library might be in different places,
such as frozen vs non-frozen applications.

[*] http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586%28v=vs.85%29.aspx

>> Ctypes only tries to find a library in the os.environ['PATH']
>> directories, not the current directory or the sys.executable directory
>> as one could expect.
>> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48
>>
>> As a workaround, for distributions that place the HDF5 and other DLLs in
>> the tables package directory, tables.__init__.py adds the tables package
>> directory to os.environ['PATH']. This also makes sure that the DLLs are
>> found when loading the hdf5Extension.pyd and other C extension modules
>> (another common problem). The use of __file__ to get the tables
>> directory should better be wrapped in a try..except statement.
>> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
>>
>> Christoph
> Thanks, I'll fix it ASAP.
>
> P.S.: please Christoph, do you have some hint for gh-175 [1]?
> There is something we can do in PyTables?
>
> [1] https://github.com/PyTables/PyTables/issues/175
Stuart

Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Stuart M. <Stu...@ob...>]

On 8/27/2012 1:22 PM, Christoph Gohlke wrote:
> On 8/27/2012 9:42 AM, Antonio Valentino wrote:
>> Hi Stuart,
>>
>> Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
>>> Hello,
>>>
>>> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:
>>>
>>>         if not ctypes.util.find_library('hdf5dll.dll'):
>>>             raise ImportError('Could not load "hdf5dll.dll", please ensure' +
>>>                     ' that it can be found in the system path')
>>>
>>> PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.
>>>
>>> That code was added in revision e9f6919.
>>>
>> It is mainly a sanity check added under request of one of our users:
>> https://github.com/PyTables/PyTables/pull/146
>>
>>
>>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.
>>>
>>> Should I file a bug report? Any easy work-around?
>>>
>>> Thanks,
>>> Stuart
>>>
>> Yes please file a pull request with your patch.
>> It would be nice to preserve the sanity check in standard case so,
>> maybe, a good solution could be adding some check on sys.frozen or
>> something like that.
>>
>> Thank you
>>
> Hello,
>
> As a workaround for frozen distributions, try to add the sys.executable
> directory to os.environ['PATH'] before importing tables.
>
> Ctypes only tries to find a library in the os.environ['PATH']
> directories, not the current directory or the sys.executable directory
> as one could expect.
> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48
>
> As a workaround, for distributions that place the HDF5 and other DLLs in
> the tables package directory, tables.__init__.py adds the tables package
> directory to os.environ['PATH']. This also makes sure that the DLLs are
> found when loading the hdf5Extension.pyd and other C extension modules
> (another common problem). The use of __file__ to get the tables
> directory should better be wrapped in a try..except statement.
> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
>
> Christoph
Hi Christoph,

Thanks for the info/suggestions. It might be nice to add your comments to the
Issue #177 I created.

I was aware that altering the PATH is a work-around. Patching PyTables is cleaner
and seems like the proper fix, and I think we agree that there is a problem here
that should be addressed. Maybe the PyTables test suite should even include a
frozen application test.

Thanks,
Stuart

Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Antonio V. <ant...@ti...>]

Hi Christoph,

thank you very much for your hints.

Il 27/08/2012 19:22, Christoph Gohlke ha scritto:
> On 8/27/2012 9:42 AM, Antonio Valentino wrote:
>> Hi Stuart,
>>
>> Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
>>> Hello,
>>>
>>> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:
>>>
>>>         if not ctypes.util.find_library('hdf5dll.dll'):
>>>             raise ImportError('Could not load "hdf5dll.dll", please ensure' +
>>>                     ' that it can be found in the system path')
>>>
>>> PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.
>>>
>>> That code was added in revision e9f6919.
>>>
>>
>> It is mainly a sanity check added under request of one of our users:
>> https://github.com/PyTables/PyTables/pull/146
>>
>>
>>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.
>>>
>>> Should I file a bug report? Any easy work-around?
>>>
>>> Thanks,
>>> Stuart
>>>
>>
>> Yes please file a pull request with your patch.
>> It would be nice to preserve the sanity check in standard case so,
>> maybe, a good solution could be adding some check on sys.frozen or
>> something like that.
>>
>> Thank you
>>
>
> Hello,
>
> As a workaround for frozen distributions, try to add the sys.executable
> directory to os.environ['PATH'] before importing tables.
>

Christoph, I suppose it can also be done at this point too:
https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
Isn't it?

Please Stuart, can you try this fix as well.

> Ctypes only tries to find a library in the os.environ['PATH']
> directories, not the current directory or the sys.executable directory
> as one could expect.
> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48
>
> As a workaround, for distributions that place the HDF5 and other DLLs in
> the tables package directory, tables.__init__.py adds the tables package
> directory to os.environ['PATH']. This also makes sure that the DLLs are
> found when loading the hdf5Extension.pyd and other C extension modules
> (another common problem). The use of __file__ to get the tables
> directory should better be wrapped in a try..except statement.
> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
>
> Christoph

Thanks, I'll fix it ASAP.

P.S.: please Christoph, do you have some hint for gh-175 [1]?
There is something we can do in PyTables?

[1] https://github.com/PyTables/PyTables/issues/175


-- 
Antonio Valentino

Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Stuart M. <Stu...@ob...>]

> Hi Stuart,
>
> Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
>> Hello,
>>
>> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:
>>
>>        if not ctypes.util.find_library('hdf5dll.dll'):
>>            raise ImportError('Could not load "hdf5dll.dll", please ensure' +
>>                    ' that it can be found in the system path')
>>
>> PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.
>>
>> That code was added in revision e9f6919.
>>
> It is mainly a sanity check added under request of one of our users:
> https://github.com/PyTables/PyTables/pull/146
>
>
>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.
>>
>> Should I file a bug report? Any easy work-around?
>>
>> Thanks,
>> Stuart
>>
> Yes please file a pull request with your patch.
> It would be nice to preserve the sanity check in standard case so,
> maybe, a good solution could be adding some check on sys.frozen or
> something like that.
>
> Thank you
Antonio,

Thanks for the quick response.

To save time I did the report as a simple Issue (https://github.com/PyTables/PyTables/issues/177) showing the patch code that preserves  and with a reference to #146. I hope that suffices.

Stuart

Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Christoph G. <cg...@uc...>]

On 8/27/2012 9:42 AM, Antonio Valentino wrote:
> Hi Stuart,
>
> Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
>> Hello,
>>
>> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:
>>
>>        if not ctypes.util.find_library('hdf5dll.dll'):
>>            raise ImportError('Could not load "hdf5dll.dll", please ensure' +
>>                    ' that it can be found in the system path')
>>
>> PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.
>>
>> That code was added in revision e9f6919.
>>
>
> It is mainly a sanity check added under request of one of our users:
> https://github.com/PyTables/PyTables/pull/146
>
>
>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.
>>
>> Should I file a bug report? Any easy work-around?
>>
>> Thanks,
>> Stuart
>>
>
> Yes please file a pull request with your patch.
> It would be nice to preserve the sanity check in standard case so,
> maybe, a good solution could be adding some check on sys.frozen or
> something like that.
>
> Thank you
>

Hello,

As a workaround for frozen distributions, try to add the sys.executable 
directory to os.environ['PATH'] before importing tables.

Ctypes only tries to find a library in the os.environ['PATH'] 
directories, not the current directory or the sys.executable directory 
as one could expect.
http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48

As a workaround, for distributions that place the HDF5 and other DLLs in 
the tables package directory, tables.__init__.py adds the tables package 
directory to os.environ['PATH']. This also makes sure that the DLLs are 
found when loading the hdf5Extension.pyd and other C extension modules 
(another common problem). The use of __file__ to get the tables 
directory should better be wrapped in a try..except statement.
https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24

Christoph

Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Antonio V. <ant...@ti...>]

Hi Stuart,

Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
> Hello,
>
> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:
>
>       if not ctypes.util.find_library('hdf5dll.dll'):
>           raise ImportError('Could not load "hdf5dll.dll", please ensure' +
>                   ' that it can be found in the system path')
>
> PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.
>
> That code was added in revision e9f6919.
>

It is mainly a sanity check added under request of one of our users:
https://github.com/PyTables/PyTables/pull/146


> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.
>
> Should I file a bug report? Any easy work-around?
>
> Thanks,
> Stuart
>

Yes please file a pull request with your patch.
It would be nice to preserve the sanity check in standard case so, 
maybe, a good solution could be adding some check on sys.frozen or 
something like that.

Thank you

-- 
Antonio Valentino

[Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Stuart M. <Stu...@ob...>]

Hello,

I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:

     if not ctypes.util.find_library('hdf5dll.dll'):
         raise ImportError('Could not load "hdf5dll.dll", please ensure' +
                 ' that it can be found in the system path')

PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.

That code was added in revision e9f6919.

This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.

Should I file a bug report? Any easy work-around?

Thanks,
Stuart

Re: [Pytables-users] Runtime warning and import error after installation

[Antonio V. <ant...@ti...>]

Hi Serena Rhie,


Il giorno 20/ago/2012, alle ore 08:40, PyTables Org <pyt...@go...> ha scritto:

> Forwarding bounced message.
> ~Josh
> 
>> From: Arang Rhie <ar...@gm...>
>> Date: August 20, 2012 3:59:06 AM GMT+01:00
>> To: pyt...@li...
>> Subject: Runtime warning and import error after installation
>> 
>> Hi,
>> 
>> I'm new to PyTable, and now getting really frustrating with the installation process.
>> 1. What is the HDF5 runtime?
>>     I've installed HDF5 following the instruction given by HDF5.
>>     What file is PyTable looking for? And how can I set the path to it?
>>     I've installed HDF5 under /usr/local/hdf5 and set the environment variable $HDF5_DIR on /usr/local/hdf5.

this is very strange. Can you please check it again and, in case, file a new issue on [1]


>>     But still setup.py is not recognizing it without the --hdf5=/usr/local/hdf5 option.
>>     Moreover,  I don't understand the 'shared library' of HDF5. does that mean to install hdf5-1.8.9-linux-x86_64-shared.tar.gz from HDF5?
>>     I've untar the ~shared.tar.gz under /usr/local/hdf5-shared and set the $LD_LIBRARY_PATH as /usr/local/hdf5-shared.

If you maintained the layout of the official tar ball then probably you should set LD_LIBRARY_PATH to point to /usr/local/hdf5-shared/lib (please note the ending "/lib").
 
>>    What else do I need to do in addition?
>> 2. I've configured LZO with --enable-shared option, and followed the instruction (make, make check, make test, make install).
>>    I've completed without any error, and the shared libraries where set under /usr/local/lib.
>>    Do I need to explicitly set other variables for running setup.py?
>> 

No, it should be OK.

If the /usr/local/lib/liblzo2.so actually exists the setup script should not complain about a missing runtime.
Can you please check and let us know.

>> 
>> 
>> [root@gmi-student tables-2.4.0]# python setup.py install --hdf5=/usr/local/hdf5
>> * Found numpy 1.6.2 package installed.
>> * Found numexpr 2.0.1 package installed.
>> * Found Cython 0.16 package installed.
>> * Found HDF5 headers at ``/usr/local/hdf5/include``, library at ``/usr/local/hdf5/lib``.
>> .. WARNING:: Could not find the HDF5 runtime.
>>    The HDF5 shared library was *not* found in the default library
>>    paths. In case of runtime problems, please remember to install it.
>> * Found LZO 2 headers at ``/usr/local/include``, library at ``/usr/local/lib``.
>> .. WARNING:: Could not find the LZO 2 runtime.
>>    The LZO 2 shared library was *not* found in the default library
>>    paths. In case of runtime problems, please remember to install it.
>> * Skipping detection of LZO 1 since LZO 2 has already been found.
>> * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib64``.
>> running install
>> running build
>> running build_py
>> running build_ext
>> running build_scripts
>> running install_lib
>> running install_scripts
>> changing mode of /usr/bin/ptdump to 755
>> changing mode of /usr/bin/ptrepack to 755
>> changing mode of /usr/bin/nctoh5 to 755
>> running install_egg_info
>> Removing /usr/lib64/python2.7/site-packages/tables-2.4.0-py2.7.egg-info
>> Writing /usr/lib64/python2.7/site-packages/tables-2.4.0-py2.7.egg-info
>> [root@gmi-student tables-2.4.0]# python
>> Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) 
>> [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import tables
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>>   File "tables/__init__.py", line 30, in <module>
>>     from tables.utilsExtension import getPyTablesVersion, getHDF5Version
>> ImportError: libhdf5.so.7: cannot open shared object file: No such file or directory
>> 
>> Serena Rhie


best regards

[1] https://github.com/PyTables/PyTables/issues

--
Antonio Valentino

[Pytables-users] Fwd: Runtime warning and import error after installation

[PyTables O. <pyt...@go...>]

Forwarding bounced message.
~Josh

> From: Arang Rhie <ar...@gm...>
> Date: August 20, 2012 3:59:06 AM GMT+01:00
> To: pyt...@li...
> Subject: Runtime warning and import error after installation
> 
> Hi,
> 
> I'm new to PyTable, and now getting really frustrating with the installation process.
> 1. What is the HDF5 runtime?
>     I've installed HDF5 following the instruction given by HDF5.
>     What file is PyTable looking for? And how can I set the path to it?
>     I've installed HDF5 under /usr/local/hdf5 and set the environment variable $HDF5_DIR on /usr/local/hdf5.
>     But still setup.py is not recognizing it without the --hdf5=/usr/local/hdf5 option.
>     Moreover,  I don't understand the 'shared library' of HDF5. does that mean to install hdf5-1.8.9-linux-x86_64-shared.tar.gz from HDF5?
>     I've untar the ~shared.tar.gz under /usr/local/hdf5-shared and set the $LD_LIBRARY_PATH as /usr/local/hdf5-shared.
>    What else do I need to do in addition?
> 2. I've configured LZO with --enable-shared option, and followed the instruction (make, make check, make test, make install).
>    I've completed without any error, and the shared libraries where set under /usr/local/lib.
>    Do I need to explicitly set other variables for running setup.py?
> 
> 
> 
> [root@gmi-student tables-2.4.0]# python setup.py install --hdf5=/usr/local/hdf5
> * Found numpy 1.6.2 package installed.
> * Found numexpr 2.0.1 package installed.
> * Found Cython 0.16 package installed.
> * Found HDF5 headers at ``/usr/local/hdf5/include``, library at ``/usr/local/hdf5/lib``.
> .. WARNING:: Could not find the HDF5 runtime.
>    The HDF5 shared library was *not* found in the default library
>    paths. In case of runtime problems, please remember to install it.
> * Found LZO 2 headers at ``/usr/local/include``, library at ``/usr/local/lib``.
> .. WARNING:: Could not find the LZO 2 runtime.
>    The LZO 2 shared library was *not* found in the default library
>    paths. In case of runtime problems, please remember to install it.
> * Skipping detection of LZO 1 since LZO 2 has already been found.
> * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib64``.
> running install
> running build
> running build_py
> running build_ext
> running build_scripts
> running install_lib
> running install_scripts
> changing mode of /usr/bin/ptdump to 755
> changing mode of /usr/bin/ptrepack to 755
> changing mode of /usr/bin/nctoh5 to 755
> running install_egg_info
> Removing /usr/lib64/python2.7/site-packages/tables-2.4.0-py2.7.egg-info
> Writing /usr/lib64/python2.7/site-packages/tables-2.4.0-py2.7.egg-info
> [root@gmi-student tables-2.4.0]# python
> Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) 
> [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import tables
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "tables/__init__.py", line 30, in <module>
>     from tables.utilsExtension import getPyTablesVersion, getHDF5Version
> ImportError: libhdf5.so.7: cannot open shared object file: No such file or directory
> 
> Serena Rhie

Re: [Pytables-users] import error

[Anthony S. <sc...@gm...>]

Hi John,

This is probably a path issue.  You likely have both pytables installed and
a 'tables' source sub-directory wherever you are running this from.  For
whatever reason, it is picking up the source version rather than the
installed version.  It is either that, or you simply don't have it
installed correctly.  The file it is missing is one which gets compiled
when you run "python setup.py install"

Be Well
Anthony

On Sat, Aug 18, 2012 at 2:28 AM, John <clo...@ya...> wrote:

> I have the following import error when I try to  import pytable
>
> ImportError                               Traceback (most recent call last)<ipython-input-2-389ecae14f10> in <module>()----> 1 import tables
> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/tables/__init__.py in <module>()     28      29 # Necessary imports to get versions stored on the Pyrex extension---> 30 from tables.utilsExtension import getPyTablesVersion, getHDF5Version     31      32
> ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/tables/utilsExtension.so, 2): Symbol not found: _H5E_CALLBACK_g
>   Referenced from: /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/tables/utilsExtension.so
>   Expected in: flat namespace
>  in /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/tables/utilsExtension.so
>
>
>
> Anyone knows whats wrong with it?
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>

[Pytables-users] import error

[John <clo...@ya...>]

I have the following import error when I try to  import pytable

ImportError Traceback (most recent call last) <ipython-input-2-389ecae14f10> in <module>() ----> 1import tables /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/tables/__init__.py in <module>() 28  29 # Necessary imports to get versions stored on the Pyrex extension ---> 30from tables.utilsExtension import getPyTablesVersion, getHDF5Version 31  32  ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/tables/utilsExtension.so, 2): Symbol not found: _H5E_CALLBACK_g Referenced from: /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/tables/utilsExtension.so Expected in: flat namespace in /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/tables/utilsExtension.so 


Anyone knows whats wrong with it?

Re: [Pytables-users] Searching for nan values in a table...

[Aquil H. A. <aqu...@gm...>]

You are correct sir there doesn't appear to be support for isnan or infinite: 

http://code.google.com/p/numexpr/issues/detail?id=23&q=nan

I'll try something far less efficient, possibly a lambda expression on table column. Anyways, thanks for root causing the issue. 

-- 
Aquil H. Abdullah
"I never think of the future. It comes soon enough" - Albert Einstein


On Thursday, August 16, 2012 at 2:01 PM, Anthony Scopatz wrote:

> So this is probably a numexpr issue.  There doesn't seem to be an isnan() implementation [1].  I would bring it up with them. Sorry we can't do more.
> 
> Be Well
> Anthony
> 
> 1. http://code.google.com/p/numexpr/wiki/UsersGuide
> 
> On Thu, Aug 16, 2012 at 12:57 PM, Aquil H. Abdullah <aqu...@gm... (mailto:aqu...@gm...)> wrote:
> > I get the same error if I use: 
> > 
> > bad_vols = tbl.getWhereList('volume == nan')
> > bad_vols = tbl.getWhereList('volume == NaN')
> > 
> > -- 
> > Aquil H. Abdullah
> > "I never think of the future. It comes soon enough" - Albert Einstein
> > 
> > 
> > On Thursday, August 16, 2012 at 1:52 PM, Anthony Scopatz wrote:
> > 
> > > Have you tried simply doing:
> > > 
> > > 'volume == nan' or
> > > 'volume == NaN'
> > > 
> > > On Thu, Aug 16, 2012 at 12:49 PM, Aquil H. Abdullah <aqu...@gm... (mailto:aqu...@gm...)> wrote:
> > > > Hello All, 
> > > > 
> > > > I am trying to determine if there are any NaN values in one of my tables, but when I queried for numpy.nan I received a NameError. Can any tell be the best way to search for a NaN value? Thanks! 
> > > > 
> > > > In [7]: type(np.nan)
> > > > Out[7]: float
> > > > 
> > > > 
> > > > In [8]: bad_vols = tbl.getWhereList('volume == %f' % np.nan)
> > > > ---------------------------------------------------------------------------
> > > > NameError                                 Traceback (most recent call last)
> > > > /Users/aquilabdullah/<ipython-input-8-2c1b183b0581> in <module>()
> > > > ----> 1 bad_vols = tbl.getWhereList('volume == %f' % np.nan)
> > > > 
> > > > /Library/Python/2.7/site-packages/tables/table.pyc in getWhereList(self, condition, condvars, sort, start, stop, step)
> > > >    1540 
> > > >    1541         coords = [ p.nrow for p in
> > > > -> 1542                    self._where(condition, condvars, start, stop, step) ]
> > > >    1543         coords = numpy.array(coords, dtype=SizeType)
> > > >    1544         # Reset the conditions
> > > > 
> > > > /Library/Python/2.7/site-packages/tables/table.pyc in _where(self, condition, condvars, start, stop, step)
> > > >    1434 
> > > >    1435         # Compile the condition and extract usable index conditions.
> > > > -> 1436         condvars = self._requiredExprVars(condition, condvars, depth=3)
> > > >    1437         compiled = self._compileCondition(condition, condvars)
> > > >    1438 
> > > > 
> > > > /Library/Python/2.7/site-packages/tables/table.pyc in _requiredExprVars(self, expression, uservars, depth) 
> > > >    1207                 val = user_globals[var]
> > > >    1208             else:
> > > > -> 1209                 raise NameError("name ``%s`` is not defined" % var)
> > > >    1210 
> > > >    1211             # Check the value.
> > > > 
> > > > NameError: name ``nan`` is not defined 
> > > > 
> > > > -- 
> > > > Aquil H. Abdullah
> > > > "I never think of the future. It comes soon enough" - Albert Einstein
> > > > 
> > > > 
> > > > ------------------------------------------------------------------------------
> > > > Live Security Virtual Conference
> > > > Exclusive live event will cover all the ways today's security and
> > > > threat landscape has changed and how IT managers can respond. Discussions
> > > > will include endpoint security, mobile security and the latest in malware
> > > > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > > > _______________________________________________
> > > > Pytables-users mailing list
> > > > Pyt...@li... (mailto:Pyt...@li...)
> > > > https://lists.sourceforge.net/lists/listinfo/pytables-users
> > > > 
> > > 
> > > ------------------------------------------------------------------------------
> > > Live Security Virtual Conference
> > > Exclusive live event will cover all the ways today's security and 
> > > threat landscape has changed and how IT managers can respond. Discussions 
> > > will include endpoint security, mobile security and the latest in malware 
> > > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > > 
> > > _______________________________________________
> > > Pytables-users mailing list
> > > Pyt...@li... (mailto:Pyt...@li...)
> > > https://lists.sourceforge.net/lists/listinfo/pytables-users
> > > 
> > > 
> > > 
> > 
> > 
> > 
> > ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li... (mailto:Pyt...@li...)
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> > 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li... (mailto:Pyt...@li...)
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 
> 

Re: [Pytables-users] Searching for nan values in a table...

[Anthony S. <sc...@gm...>]

So this is probably a numexpr issue.  There doesn't seem to be an isnan()
implementation [1].  I would bring it up with them. Sorry we can't do more.

Be Well
Anthony

1. http://code.google.com/p/numexpr/wiki/UsersGuide

On Thu, Aug 16, 2012 at 12:57 PM, Aquil H. Abdullah <
aqu...@gm...> wrote:

> I get the same error if I use:
>
> bad_vols = tbl.getWhereList('volume == nan')
> bad_vols = tbl.getWhereList('volume == NaN')
>
> --
> Aquil H. Abdullah
> "I never think of the future. It comes soon enough" - Albert Einstein
>
> On Thursday, August 16, 2012 at 1:52 PM, Anthony Scopatz wrote:
>
> Have you tried simply doing:
>
> 'volume == nan' or
> 'volume == NaN'
>
> On Thu, Aug 16, 2012 at 12:49 PM, Aquil H. Abdullah <
> aqu...@gm...> wrote:
>
>  Hello All,
>
> I am trying to determine if there are any NaN values in one of my tables,
> but when I queried for numpy.nan I received a NameError. Can any tell be
> the best way to search for a NaN value? Thanks!
>
> In [7]: type(np.nan)
> Out[7]: float
>
> In [8]: bad_vols = tbl.getWhereList('volume == %f' % np.nan)
> ---------------------------------------------------------------------------
> NameError                                 Traceback (most recent call last)
> /Users/aquilabdullah/<ipython-input-8-2c1b183b0581> in <module>()
> ----> 1 bad_vols = tbl.getWhereList('volume == %f' % np.nan)
>
> /Library/Python/2.7/site-packages/tables/table.pyc in getWhereList(self,
> condition, condvars, sort, start, stop, step)
>    1540
>    1541         coords = [ p.nrow for p in
> -> 1542                    self._where(condition, condvars, start, stop,
> step) ]
>    1543         coords = numpy.array(coords, dtype=SizeType)
>    1544         # Reset the conditions
>
> /Library/Python/2.7/site-packages/tables/table.pyc in _where(self,
> condition, condvars, start, stop, step)
>    1434
>    1435         # Compile the condition and extract usable index
> conditions.
> -> 1436         condvars = self._requiredExprVars(condition, condvars,
> depth=3)
>    1437         compiled = self._compileCondition(condition, condvars)
>    1438
>
> /Library/Python/2.7/site-packages/tables/table.pyc in
> _requiredExprVars(self, expression, uservars, depth)
>    1207                 val = user_globals[var]
>    1208             else:
> -> 1209                 raise NameError("name ``%s`` is not defined" % var)
>    1210
>    1211             # Check the value.
>
> NameError: name ``nan`` is not defined
>
> --
> Aquil H. Abdullah
> "I never think of the future. It comes soon enough" - Albert Einstein
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>

Re: [Pytables-users] Searching for nan values in a table...

[Aquil H. A. <aqu...@gm...>]

I get the same error if I use: 

bad_vols = tbl.getWhereList('volume == nan')
bad_vols = tbl.getWhereList('volume == NaN')

-- 
Aquil H. Abdullah
"I never think of the future. It comes soon enough" - Albert Einstein


On Thursday, August 16, 2012 at 1:52 PM, Anthony Scopatz wrote:

> Have you tried simply doing:
> 
> 'volume == nan' or
> 'volume == NaN'
> 
> On Thu, Aug 16, 2012 at 12:49 PM, Aquil H. Abdullah <aqu...@gm... (mailto:aqu...@gm...)> wrote:
> > Hello All, 
> > 
> > I am trying to determine if there are any NaN values in one of my tables, but when I queried for numpy.nan I received a NameError. Can any tell be the best way to search for a NaN value? Thanks! 
> > 
> > In [7]: type(np.nan)
> > Out[7]: float
> > 
> > 
> > In [8]: bad_vols = tbl.getWhereList('volume == %f' % np.nan)
> > ---------------------------------------------------------------------------
> > NameError                                 Traceback (most recent call last)
> > /Users/aquilabdullah/<ipython-input-8-2c1b183b0581> in <module>()
> > ----> 1 bad_vols = tbl.getWhereList('volume == %f' % np.nan)
> > 
> > /Library/Python/2.7/site-packages/tables/table.pyc in getWhereList(self, condition, condvars, sort, start, stop, step)
> >    1540 
> >    1541         coords = [ p.nrow for p in
> > -> 1542                    self._where(condition, condvars, start, stop, step) ]
> >    1543         coords = numpy.array(coords, dtype=SizeType)
> >    1544         # Reset the conditions
> > 
> > /Library/Python/2.7/site-packages/tables/table.pyc in _where(self, condition, condvars, start, stop, step)
> >    1434 
> >    1435         # Compile the condition and extract usable index conditions.
> > -> 1436         condvars = self._requiredExprVars(condition, condvars, depth=3)
> >    1437         compiled = self._compileCondition(condition, condvars)
> >    1438 
> > 
> > /Library/Python/2.7/site-packages/tables/table.pyc in _requiredExprVars(self, expression, uservars, depth) 
> >    1207                 val = user_globals[var]
> >    1208             else:
> > -> 1209                 raise NameError("name ``%s`` is not defined" % var)
> >    1210 
> >    1211             # Check the value.
> > 
> > NameError: name ``nan`` is not defined 
> > 
> > -- 
> > Aquil H. Abdullah
> > "I never think of the future. It comes soon enough" - Albert Einstein
> > 
> > 
> > ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li... (mailto:Pyt...@li...)
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> > 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li... (mailto:Pyt...@li...)
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 
> 

Re: [Pytables-users] Searching for nan values in a table...

[Anthony S. <sc...@gm...>]

Have you tried simply doing:

'volume == nan' or
'volume == NaN'

On Thu, Aug 16, 2012 at 12:49 PM, Aquil H. Abdullah <
aqu...@gm...> wrote:

>  Hello All,
>
> I am trying to determine if there are any NaN values in one of my tables,
> but when I queried for numpy.nan I received a NameError. Can any tell be
> the best way to search for a NaN value? Thanks!
>
> In [7]: type(np.nan)
> Out[7]: float
>
> In [8]: bad_vols = tbl.getWhereList('volume == %f' % np.nan)
> ---------------------------------------------------------------------------
> NameError                                 Traceback (most recent call last)
> /Users/aquilabdullah/<ipython-input-8-2c1b183b0581> in <module>()
> ----> 1 bad_vols = tbl.getWhereList('volume == %f' % np.nan)
>
> /Library/Python/2.7/site-packages/tables/table.pyc in getWhereList(self,
> condition, condvars, sort, start, stop, step)
>    1540
>    1541         coords = [ p.nrow for p in
> -> 1542                    self._where(condition, condvars, start, stop,
> step) ]
>    1543         coords = numpy.array(coords, dtype=SizeType)
>    1544         # Reset the conditions
>
> /Library/Python/2.7/site-packages/tables/table.pyc in _where(self,
> condition, condvars, start, stop, step)
>    1434
>    1435         # Compile the condition and extract usable index
> conditions.
> -> 1436         condvars = self._requiredExprVars(condition, condvars,
> depth=3)
>    1437         compiled = self._compileCondition(condition, condvars)
>    1438
>
> /Library/Python/2.7/site-packages/tables/table.pyc in
> _requiredExprVars(self, expression, uservars, depth)
>    1207                 val = user_globals[var]
>    1208             else:
> -> 1209                 raise NameError("name ``%s`` is not defined" % var)
>    1210
>    1211             # Check the value.
>
> NameError: name ``nan`` is not defined
>
> --
> Aquil H. Abdullah
> "I never think of the future. It comes soon enough" - Albert Einstein
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>

[Pytables-users] Searching for nan values in a table...

[Aquil H. A. <aqu...@gm...>]

Hello All, 

I am trying to determine if there are any NaN values in one of my tables, but when I queried for numpy.nan I received a NameError. Can any tell be the best way to search for a NaN value? Thanks!

In [7]: type(np.nan)
Out[7]: float


In [8]: bad_vols = tbl.getWhereList('volume == %f' % np.nan)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/Users/aquilabdullah/<ipython-input-8-2c1b183b0581> in <module>()
----> 1 bad_vols = tbl.getWhereList('volume == %f' % np.nan)

/Library/Python/2.7/site-packages/tables/table.pyc in getWhereList(self, condition, condvars, sort, start, stop, step)
   1540 
   1541         coords = [ p.nrow for p in
-> 1542                    self._where(condition, condvars, start, stop, step) ]
   1543         coords = numpy.array(coords, dtype=SizeType)
   1544         # Reset the conditions

/Library/Python/2.7/site-packages/tables/table.pyc in _where(self, condition, condvars, start, stop, step)
   1434 
   1435         # Compile the condition and extract usable index conditions.
-> 1436         condvars = self._requiredExprVars(condition, condvars, depth=3)
   1437         compiled = self._compileCondition(condition, condvars)
   1438 

/Library/Python/2.7/site-packages/tables/table.pyc in _requiredExprVars(self, expression, uservars, depth)
   1207                 val = user_globals[var]
   1208             else:
-> 1209                 raise NameError("name ``%s`` is not defined" % var)
   1210 
   1211             # Check the value.

NameError: name ``nan`` is not defined 

-- 
Aquil H. Abdullah
"I never think of the future. It comes soon enough" - Albert Einstein

Re: [Pytables-users] openFile strategy question

[Andre' Walker-L. <wal...@gm...>]

Hi Anthony,

> Oh OK, I think I understand a little better. What I would do would be to make "for i,file in enumerate(hdf5_files)" the outer most loop and then use the File.walkNodes() method [1] to walk each file and pick out only the data sets that you want to copy, skipping over all others.  This should allow you to only open each of the 400 files once.  Hope this helps.

Thanks.  This is the idea I had, but was failing to implement (although I didn't use walkNodes).  To get it to work, I had to figure out how to use createEArray properly.  In the end, it was a silly fix.

I created an EArray with shape (0,96,1,2), and was trying to append numpy arrays of shape (96,1,2) to this, which was failing.  In the end, all I had to do was

arr.append(np.array([my_array]))

where as before, I was simply missing the "[ ]" brackets, so the shapes did not line up.


Cheers,

Andre

Re: [Pytables-users] In-kernal for subset?

[Adam D. <ade...@ex...>]

From: Anthony Scopatz <sc...@gm...<mailto:sc...@gm...>>
Reply-To: Discussion list for PyTables <pyt...@li...<mailto:pyt...@li...>>
Date: Wednesday, August 15, 2012 11:29 PM
To: Discussion list for PyTables <pyt...@li...<mailto:pyt...@li...>>
Subject: Re: [Pytables-users] In-kernal for subset?

On Thu, Aug 16, 2012 at 1:06 AM, Adam Dershowitz <ade...@ex...<mailto:ade...@ex...>> wrote:
From: Anthony Scopatz <sc...@gm...<mailto:sc...@gm...>>
Reply-To: Discussion list for PyTables <pyt...@li...<mailto:pyt...@li...>>
Date: Wednesday, August 15, 2012 2:47 PM
To: Discussion list for PyTables <pyt...@li...<mailto:pyt...@li...>>
Subject: Re: [Pytables-users] In-kernal for subset?

On Wed, Aug 15, 2012 at 12:33 PM, Adam Dershowitz <ade...@ex...<mailto:ade...@ex...>> wrote:
I am trying to find all cases where a value transitions above a threshold.  So, my code first does a getwherelist to find values that are above the threshold, then it uses that list to find immediately prior values that are below.  The code is working, but the second part, searching through just a smaller subset is much slower (First search is on the order of 1 second, while the second is a minute).
Is there any way to get this second part of the search in-kernal?  Or any more general way to do a search for values above a threshold, where the prior value is below?
Essentially, what I am looking for is a way to speed up that second search for "all rows in a prior defined list, where a condition is applied to the table"

My table is just seconds and values, in chronological order.

Here is the code that I am using now:

h5data = tb.openFile("AllData.h5","r")
table1 = h5data.root.table1

#Find all values above threshold:
thelist= table1.getWhereList("""Value > 150""")

#From the above list find all values where the immediately prior value is below:
transition=[]
for i in thelist:
if (table1[i-1]['Value'] < 150) and (i != 0) :
transition.append(i)

Hey Adam,

Sorry for taking a while to respond.  Assuming you don't mind one of these being <= or >=, you don't really need the second loop with a little index arithmetic:

import numpy as np
inds = np.array(thelist)
dinds = inds[1:] - inds[:-1]
transition = dinds[(1 < dinds)]

This should get you an array of all of the transition indices since wherever the difference in indices is greater than 1 the Value must have dropped below the threshold and then returned back up.

Be Well
Anthony



Thanks much for the response.  At first it didn't work, but it gave me the right idea, and now I got it working.  There were two problems above.  1)  I believe that yon u had a typo and the last line should have been "inds[(1 < …" and not "dinds[(1<…"  Otherwise you just get back the deltas instead of the actual index values.

Whoops, serves me right for hacking this out so quickly!

But, that still returned an array that wasn't working.  Turns out, after thinking some, that it was actually offset by one.  So by prepending a value into dinds (greater then 1, since the first value greater than the threshold, must always be a transition or the first table entry) it seems to solve the problem.  Here is the code that seems to work:

import numpy as np
inds = np.array(thelist)
dinds=np.append([2],inds[1:] - inds[:-1])
trans=inds[(1<dinds)]

Now, I am still curious, more for academic reasons, since the code now works, if there would be a way to speed up the second loop above?  It seems like there are other examples, where index arithmetic might not work, so is there a way to do an in-kernal search through just a subset of a table?

So the issue is that we rely on numexpr here for our in-kernel queries and numpexpr doesn't support indexing at all.  There may be hope for this in the future (see numba).  So the go stndexal here is to do whatever you can to not have queries which rely on comparing two different indexes of the same data.

If you really wanted to do this quickly and in kernel, you could probably store two copies of the data. Call 'a' the original and 'b' a copy of 'a' that is offset by 1 index and has a dummy value at the end (to make them the same size).  Then you could do something like:

tb.Expr('a == b')

This would only work on Array, CArray, and Earray data.  You might be able to get it to work using Tables with something like:

tb.Expr('a == b', uservars={'a': atable, 'b': btable})

I hope this helps.
Be Well
Anthony



Yes, it helps explain the issue.  I appreciate the info.

--Adam

Re: [Pytables-users] In-kernal for subset?

[Anthony S. <sc...@gm...>]

On Thu, Aug 16, 2012 at 1:06 AM, Adam Dershowitz
<ade...@ex...>wrote:

>   From: Anthony Scopatz <sc...@gm...>
> Reply-To: Discussion list for PyTables <
> pyt...@li...>
> Date: Wednesday, August 15, 2012 2:47 PM
> To: Discussion list for PyTables <pyt...@li...>
> Subject: Re: [Pytables-users] In-kernal for subset?
>
>   On Wed, Aug 15, 2012 at 12:33 PM, Adam Dershowitz <
> ade...@ex...> wrote:
>
>>  I am trying to find all cases where a value transitions above a
>> threshold.  So, my code first does a getwherelist to find values that are
>> above the threshold, then it uses that list to find immediately prior
>> values that are below.  The code is working, but the second part, searching
>> through just a smaller subset is much slower (First search is on the order
>> of 1 second, while the second is a minute).
>> Is there any way to get this second part of the search in-kernal?  Or any
>> more general way to do a search for values above a threshold, where the
>> prior value is below?
>> Essentially, what I am looking for is a way to speed up that second
>> search for "all rows in a prior defined list, where a condition is applied
>> to the table"
>>
>>  My table is just seconds and values, in chronological order.
>>
>>  Here is the code that I am using now:
>>
>>  h5data = tb.openFile("AllData.h5","r")
>> table1 = h5data.root.table1
>>
>>  #Find all values above threshold:
>>  thelist= table1.getWhereList("""Value > 150""")
>>
>>  #From the above list find all values where the immediately prior value
>> is below:
>> transition=[]
>> for i in thelist:
>> if (table1[i-1]['Value'] < 150) and (i != 0) :
>> transition.append(i)
>>
>
>  Hey Adam,
>
>  Sorry for taking a while to respond.  Assuming you don't mind one of
> these being <= or >=, you don't really need the second loop with a little
> index arithmetic:
>
>  import numpy as np
> inds = np.array(thelist)
> dinds = inds[1:] - inds[:-1]
> transition = dinds[(1 < dinds)]
>
>  This should get you an array of all of the transition indices since
> wherever the difference in indices is greater than 1 the Value must have
> dropped below the threshold and then returned back up.
>
>  Be Well
> Anthony
>
>
>>
>>
>  Thanks much for the response.  At first it didn't work, but it gave me
> the right idea, and now I got it working.  There were two problems above.
>  1)  I believe that yon u had a typo and the last line should have been
> "inds[(1 < …" and not "dinds[(1<…"  Otherwise you just get back the deltas
> instead of the actual index values.
>

Whoops, serves me right for hacking this out so quickly!


>  But, that still returned an array that wasn't working.  Turns out, after
> thinking some, that it was actually offset by one.  So by prepending a
> value into dinds (greater then 1, since the first value greater than the
> threshold, must always be a transition or the first table entry) it seems
> to solve the problem.  Here is the code that seems to work:
>
>  import numpy as np
>  inds = np.array(thelist)
> dinds=np.append([2],inds[1:] - inds[:-1])
> trans=inds[(1<dinds)]
>
>  Now, I am still curious, more for academic reasons, since the code now
> works, if there would be a way to speed up the second loop above?  It seems
> like there are other examples, where index arithmetic might not work, so is
> there a way to do an in-kernal search through just a subset of a table?
>

So the issue is that we rely on numexpr here for our in-kernel queries and
numpexpr doesn't support indexing at all.  There may be hope for this in
the future (see numba).  So the go stndexal here is to do whatever you can
to not have queries which rely on comparing two different indexes of the
same data.

If you really wanted to do this quickly and in kernel, you could probably
store two copies of the data. Call 'a' the original and 'b' a copy of 'a'
that is offset by 1 index and has a dummy value at the end (to make them
the same size).  Then you could do something like:

tb.Expr('a == b')

This would only work on Array, CArray, and Earray data.  You might be able
to get it to work using Tables with something like:

tb.Expr('a == b', uservars={'a': atable, 'b': btable})

I hope this helps.
Be Well
Anthony


>  Again, thanks for the help!
>
>  Best,
>
>  --Adam
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>

Re: [Pytables-users] In-kernal for subset?

[Adam D. <ade...@ex...>]

From: Anthony Scopatz <sc...@gm...<mailto:sc...@gm...>>
Reply-To: Discussion list for PyTables <pyt...@li...<mailto:pyt...@li...>>
Date: Wednesday, August 15, 2012 2:47 PM
To: Discussion list for PyTables <pyt...@li...<mailto:pyt...@li...>>
Subject: Re: [Pytables-users] In-kernal for subset?

On Wed, Aug 15, 2012 at 12:33 PM, Adam Dershowitz <ade...@ex...<mailto:ade...@ex...>> wrote:
I am trying to find all cases where a value transitions above a threshold.  So, my code first does a getwherelist to find values that are above the threshold, then it uses that list to find immediately prior values that are below.  The code is working, but the second part, searching through just a smaller subset is much slower (First search is on the order of 1 second, while the second is a minute).
Is there any way to get this second part of the search in-kernal?  Or any more general way to do a search for values above a threshold, where the prior value is below?
Essentially, what I am looking for is a way to speed up that second search for "all rows in a prior defined list, where a condition is applied to the table"

My table is just seconds and values, in chronological order.

Here is the code that I am using now:

h5data = tb.openFile("AllData.h5","r")
table1 = h5data.root.table1

#Find all values above threshold:
thelist= table1.getWhereList("""Value > 150""")

#From the above list find all values where the immediately prior value is below:
transition=[]
for i in thelist:
if (table1[i-1]['Value'] < 150) and (i != 0) :
transition.append(i)

Hey Adam,

Sorry for taking a while to respond.  Assuming you don't mind one of these being <= or >=, you don't really need the second loop with a little index arithmetic:

import numpy as np
inds = np.array(thelist)
dinds = inds[1:] - inds[:-1]
transition = dinds[(1 < dinds)]

This should get you an array of all of the transition indices since wherever the difference in indices is greater than 1 the Value must have dropped below the threshold and then returned back up.

Be Well
Anthony



Thanks much for the response.  At first it didn't work, but it gave me the right idea, and now I got it working.  There were two problems above.  1)  I believe that you had a typo and the last line should have been "inds[(1 < …" and not "dinds[(1<…"  Otherwise you just get back the deltas instead of the actual index values.
But, that still returned an array that wasn't working.  Turns out, after thinking some, that it was actually offset by one.  So by prepending a value into dinds (greater then 1, since the first value greater than the threshold, must always be a transition or the first table entry) it seems to solve the problem.  Here is the code that seems to work:

import numpy as np
inds = np.array(thelist)
dinds=np.append([2],inds[1:] - inds[:-1])
trans=inds[(1<dinds)]

Now, I am still curious, more for academic reasons, since the code now works, if there would be a way to speed up the second loop above?  It seems like there are other examples, where index arithmetic might not work, so is there a way to do an in-kernal search through just a subset of a table?

Again, thanks for the help!

Best,

--Adam

Re: [Pytables-users] openFile strategy question

[Anthony S. <sc...@gm...>]

Hi Andre,

Oh OK, I think I understand a little better. What I would do would be to
make "for i,file in enumerate(hdf5_files)" the outer most loop and then use
the File.walkNodes() method [1] to walk each file and pick out only the
data sets that you want to copy, skipping over all others.  This should
allow you to only open each of the 400 files once.  Hope this helps.

Be Well
Anthony

1.
http://pytables.github.com/usersguide/libref/file_class.html?highlight=walknodes#tables.File.walkNodes

On Wed, Aug 15, 2012 at 7:25 PM, Andre' Walker-Loud <wal...@gm...>wrote:

> Hi Anthony,
>
> > I am a little confused.  Let me verify.  You have 400 hdf5 file (re and
> im) buried in an a unix directory tree.  You want to make a single file
> which concatenates this data.  Is this right?
>
> Sorry for my description - that is not quite right.
> The "unix directory tree" is the group tree I have made in each individual
> hdf5 file.  So I have 400 hdf5 files, each with the given directory tree.
>  And I basically want to copy that directory tree, but "merge" all of them
> together.
> However, there are bits in each of the small files that I do not want to
> merge - I only want to grab the average data sets, while the little files
> contains many different samples (which I have already averaged into the
> "avg" group.
>
> Is this clear?
>
>
> Thanks,
>
> Andre
>
>
>
> >
> > Be Well
> > Anthony
> >
> > On Wed, Aug 15, 2012 at 6:52 PM, Andre' Walker-Loud <wal...@gm...>
> wrote:
> > Hi All,
> >
> > Just a strategy question.
> > I have many hdf5 files containing data for different measurements of the
> same quantities.
> >
> > My directory tree looks like
> >
> > top description [ group ]
> >   sub description [ group ]
> >     avg [ group ]
> >       re [ numpy array shape = (96,1,2) ]
> >       im [ numpy array shape = (96,1,2) ] - only exists for know subset
> of data files
> >
> > I have ~400 of these files.  What I want to do is create a single file,
> which collects all of these files with exactly the same directory
> structure, except at the very bottom
> >
> >       re [ numpy array shape = (400,96,1,2) ]
> >
> >
> > The simplest thing I came up with to do this is loop over the two levels
> of descriptive group structures, and build the numpy array for the final
> set this way.
> >
> > basic loop structure:
> >
> > final_file = tables.openFile('all_data.h5','a')
> >
> > for d1 in top_description:
> >     final_file.createGroup(final_file.root,d1)
> >     for d2 in sub_description:
> >         final_file.createGroup(final_file.root+'/'+d1,d2)
> >         data_re = np.zeros([400,96,1,2])
> >         for i,file in enumerate(hdf5_files):
> >             tmp = tables.openFile(file)
> >             data_re[i] = np.array(tmp.getNode('/d1/d2/avg/re')
> >             tmp.close()
> >
> final_file.createArray(final_file.root+'/'+d1+'/'+d2,'re',data_re)
> >
> >
> > But this involves opening and closing the individual 400 hdf5 files many
> times.
> > There must be a smarter algorithmic way to do this - or perhaps built in
> pytables tools.
> >
> > Any advice is appreciated.
> >
> >
> > Andre
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li...
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
> > Pytables-users mailing list
> > Pyt...@li...
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

Re: [Pytables-users] openFile strategy question

[Andre' Walker-L. <wal...@gm...>]

Hi Anthony,

> I am a little confused.  Let me verify.  You have 400 hdf5 file (re and im) buried in an a unix directory tree.  You want to make a single file which concatenates this data.  Is this right?

Sorry for my description - that is not quite right.
The "unix directory tree" is the group tree I have made in each individual hdf5 file.  So I have 400 hdf5 files, each with the given directory tree.  And I basically want to copy that directory tree, but "merge" all of them together.
However, there are bits in each of the small files that I do not want to merge - I only want to grab the average data sets, while the little files contains many different samples (which I have already averaged into the "avg" group.

Is this clear?


Thanks,

Andre



> 
> Be Well
> Anthony
> 
> On Wed, Aug 15, 2012 at 6:52 PM, Andre' Walker-Loud <wal...@gm...> wrote:
> Hi All,
> 
> Just a strategy question.
> I have many hdf5 files containing data for different measurements of the same quantities.
> 
> My directory tree looks like
> 
> top description [ group ]
>   sub description [ group ]
>     avg [ group ]
>       re [ numpy array shape = (96,1,2) ]
>       im [ numpy array shape = (96,1,2) ] - only exists for know subset of data files
> 
> I have ~400 of these files.  What I want to do is create a single file, which collects all of these files with exactly the same directory structure, except at the very bottom
> 
>       re [ numpy array shape = (400,96,1,2) ]
> 
> 
> The simplest thing I came up with to do this is loop over the two levels of descriptive group structures, and build the numpy array for the final set this way.
> 
> basic loop structure:
> 
> final_file = tables.openFile('all_data.h5','a')
> 
> for d1 in top_description:
>     final_file.createGroup(final_file.root,d1)
>     for d2 in sub_description:
>         final_file.createGroup(final_file.root+'/'+d1,d2)
>         data_re = np.zeros([400,96,1,2])
>         for i,file in enumerate(hdf5_files):
>             tmp = tables.openFile(file)
>             data_re[i] = np.array(tmp.getNode('/d1/d2/avg/re')
>             tmp.close()
>         final_file.createArray(final_file.root+'/'+d1+'/'+d2,'re',data_re)
> 
> 
> But this involves opening and closing the individual 400 hdf5 files many times.
> There must be a smarter algorithmic way to do this - or perhaps built in pytables tools.
> 
> Any advice is appreciated.
> 
> 
> Andre
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] openFile strategy question

[Anthony S. <sc...@gm...>]

Hi Andre,

I am a little confused.  Let me verify.  You have 400 hdf5 file (re and
im) buried in an a unix directory tree.  You want to make a single file
which concatenates this data.  Is this right?

Be Well
Anthony

On Wed, Aug 15, 2012 at 6:52 PM, Andre' Walker-Loud <wal...@gm...>wrote:

> Hi All,
>
> Just a strategy question.
> I have many hdf5 files containing data for different measurements of the
> same quantities.
>
> My directory tree looks like
>
> top description [ group ]
>   sub description [ group ]
>     avg [ group ]
>       re [ numpy array shape = (96,1,2) ]
>       im [ numpy array shape = (96,1,2) ] - only exists for know subset of
> data files
>
> I have ~400 of these files.  What I want to do is create a single file,
> which collects all of these files with exactly the same directory
> structure, except at the very bottom
>
>       re [ numpy array shape = (400,96,1,2) ]
>
>
> The simplest thing I came up with to do this is loop over the two levels
> of descriptive group structures, and build the numpy array for the final
> set this way.
>
> basic loop structure:
>
> final_file = tables.openFile('all_data.h5','a')
>
> for d1 in top_description:
>     final_file.createGroup(final_file.root,d1)
>     for d2 in sub_description:
>         final_file.createGroup(final_file.root+'/'+d1,d2)
>         data_re = np.zeros([400,96,1,2])
>         for i,file in enumerate(hdf5_files):
>             tmp = tables.openFile(file)
>             data_re[i] = np.array(tmp.getNode('/d1/d2/avg/re')
>             tmp.close()
>         final_file.createArray(final_file.root+'/'+d1+'/'+d2,'re',data_re)
>
>
> But this involves opening and closing the individual 400 hdf5 files many
> times.
> There must be a smarter algorithmic way to do this - or perhaps built in
> pytables tools.
>
> Any advice is appreciated.
>
>
> Andre
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

[Pytables-users] openFile strategy question

[Andre' Walker-L. <wal...@gm...>]

Hi All,

Just a strategy question.
I have many hdf5 files containing data for different measurements of the same quantities.

My directory tree looks like

top description [ group ]
  sub description [ group ]
    avg [ group ]
      re [ numpy array shape = (96,1,2) ]
      im [ numpy array shape = (96,1,2) ] - only exists for know subset of data files

I have ~400 of these files.  What I want to do is create a single file, which collects all of these files with exactly the same directory structure, except at the very bottom

      re [ numpy array shape = (400,96,1,2) ]


The simplest thing I came up with to do this is loop over the two levels of descriptive group structures, and build the numpy array for the final set this way.

basic loop structure:

final_file = tables.openFile('all_data.h5','a')

for d1 in top_description:
    final_file.createGroup(final_file.root,d1)
    for d2 in sub_description:
        final_file.createGroup(final_file.root+'/'+d1,d2)
        data_re = np.zeros([400,96,1,2])
        for i,file in enumerate(hdf5_files):
            tmp = tables.openFile(file)
            data_re[i] = np.array(tmp.getNode('/d1/d2/avg/re')
            tmp.close()
        final_file.createArray(final_file.root+'/'+d1+'/'+d2,'re',data_re)


But this involves opening and closing the individual 400 hdf5 files many times.
There must be a smarter algorithmic way to do this - or perhaps built in pytables tools.

Any advice is appreciated.


Andre