Re: [Pytables-users] Multithreaded decompress unexpectedly does not help

[Alvaro T. C. <al...@mi...>]

I have now similar dependencies as you, except for Numpy 1.7 beta 2.

I wish I could help with the carray flavor.

--
  Running setup.py install for carray
    * Found Cython 0.17.2 package installed.
    * Found numpy 1.6.2 package installed.
    * Found numexpr 2.0.1 package installed.
    building 'carray.carrayExtension' extension
    C compiler: gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv
-DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
-D_GNU_SOURCE -fPIC -fwrapv -fPIC

    compile options: '-Iblosc
-I/home/tejero/Local/Envs/test/lib/python2.7/site-packages/numpy/core/include
-I/usr/include/python2.7 -c'
    extra options: '-msse2'
    gcc: blosc/blosclz.c
    gcc: carray/carrayExtension.c
    gcc: error: carray/carrayExtension.c: No such file or directory
    gcc: fatal error: no input files
    compilation terminated.
    gcc: error: carray/carrayExtension.c: No such file or directory
    gcc: fatal error: no input files
    compilation terminated.
    error: Command "gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv
-DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
-D_GNU_SOURCE -fPIC -fwrapv -fPIC -Iblosc
-I/home/tejero/Local/Envs/test/lib/python2.7/site-packages/numpy/core/include
-I/usr/include/python2.7 -c carray/carrayExtension.c -o
build/temp.linux-x86_64-2.7/carray/carrayExtension.o -msse2" failed with
exit status 4



-á.



On 7 December 2012 12:47, Francesc Alted <fa...@gm...> wrote:

> On 12/6/12 1:42 PM, Alvaro Tejero Cantero wrote:
> > Thank you for the comprehensive round-up. I have some ideas and
> > reports below.
> >
> > What about ctables? The documentation says that it is specificly
> > column-access optimized, which is what I need in this scenario
> > (sometimes sequential, sometimes random).
>
> Yes, ctables is optimized for column access.
>
> >
> > Unfortunately I could not get the rootdir parameter for ctables
> > __init__ to work in carray 0.4 and pip-installing 0.5 or 0.5.1 leads
> > to compilation errors.
>
> Yep, persistence for carray/ctables objects was added in 0.5.
>
> >
> > This is the ctables-to-disk error:
> >
> > ct2 = ca.ctable((np.arange(30000000),), names=('range2',),
> > rootdir='/tmp/ctable2.ctable')
> >
> ---------------------------------------------------------------------------
> > TypeError                                  Traceback (most recent call
> last)
> > /home/tejero/Dropbox/O/nb/nonridge/<ipython-input-29-255842877a0b>
>  in<module>()
> > ----> 1  ct2=  ca.ctable((np.arange(30000000),),  names=('range2',),
>  rootdir='/tmp/ctable2.ctable')
> >
> >
> /home/tejero/Local/Envs/test/lib/python2.7/site-packages/carray/ctable.pyc
>  in__init__(self, cols, names, **kwargs)
> >      158                  if  column.dtype==  np.void:
> >      159                      raise  ValueError,  "`cols` elements
> cannot be of type void"
> > --> 160                  column=  ca.carray(column,  **kwargs)
> >      161              elif  ratype:
> >      162                  column=  ca.carray(cols[name],  **kwargs)
> >
> >
> /home/tejero/Local/Envs/test/lib/python2.7/site-packages/carray/carrayExtension.so
>  incarray.carrayExtension.carray.__cinit__ (carray/carrayExtension.c:3917)()
> >
> > TypeError: __cinit__() got an unexpected keyword argument 'rootdir'
> >
> >
> > And this is cut from the pip output when trying to upgrade carray.
> >
> > gcc: carray/carrayExtension.c
> >
> > gcc: error: carray/carrayExtension.c: No such file or directory
>
> Hmm, that's strange, because the carrayExtension should have been
> cythonized automatically.  Here it is part of my install process with pip:
>
>    Running setup.py install for carray
>      * Found Cython 0.17.1 package installed.
>      * Found numpy 1.7.0b2 package installed.
>      * Found numexpr 2.0.1 package installed.
>      cythoning carray/carrayExtension.pyx to carray/carrayExtension.c
>      building 'carray.carrayExtension' extension
>      C compiler: gcc -fno-strict-aliasing
> -I/Users/faltet/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes
>
> Hmm, perhaps you need a newer version of Cython?
>
> >
> >
> > Two more notes:
> >
> > * a way was added to check in-disk (compressed) vs in-memory
> > (uncompressed) node sizes. I was unable to find the way to use it
> > either from the 2.4.0 release notes or from the git issue
> > https://github.com/PyTables/PyTables/issues/141#issuecomment-5018763
>
> You already found the answer.
>
> >
> > * is/will it be possible to load PyTables carrays as in-memory carrays
> > without decompression?
>
> Actually, that has been my idea from the very beginning.  The concept of
> 'flavor' for the returned objects when reading is already there, so it
> should be relatively easy to add a new 'carray' flavor.  Maybe you can
> contribute this?
>
> --
> Francesc Alted
>
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>