I've been unable to build pydoop on my Centos6.2 machine via any method (pip, easy_install, or download and build). The version I am trying to install is 0.10.0. There are many build errors associated with include files, along the lines of:
In file included from /usr/include/python2.6/Python.h:42,
from /usr/include/boost/python/detail/wrap_python.hpp:142,
from /usr/include/boost/python/detail/prefix.hpp:13,
from /usr/include/boost/python/args.hpp:8,
from /usr/include/boost/python.hpp:11,
from src/hdfs_common.hpp:31,
from src/hdfs_fs.hpp:24,
from src/hdfs_fs.cpp:21:
/usr/include/stdlib.h:503: error: ‘size_t’ was not declared in this scope
/usr/include/stdlib.h:503: error: expected ‘,’ or ‘;’ before ‘throw’
/usr/include/stdlib.h:508: error: ‘size_t’ has not been declared
and on and on and on for pages . . .
Has anyone else encountered this and conquered it?
p.s. See blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/ for another user with similar build issues.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Why are only the older versions of CDH 4.X supported? Is there any way to get it to work with CDH 4.5? I have hadoop 2.0.0-cdh4.5.0 and am trying to install pydoop 0.11.1.
Ensuing errors are:
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:39:26: error: openssl/hmac.h: No such file or directory
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:40:28: error: openssl/buffer.h: No such file or directory
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc: In member function ‘std::string HadoopPipes::BinaryProtocol::createDigest(std::string&, std::string&)’:
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:423: error: ‘HMAC_CTX’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:423: error: expected ‘;’ before ‘ctx’
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:424: error: ‘EVP_MAX_MD_SIZE’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:425: error: ‘ctx’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:426: error: ‘EVP_sha1’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:426: error: ‘HMAC_Init’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:427: error: ‘HMAC_Update’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:429: error: ‘digest’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:429: error: ‘HMAC_Final’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:430: error: ‘HMAC_cleanup’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:433: error: ‘BIO’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:433: error: ‘bmem’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:433: error: ‘b64’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:434: error: ‘BUF_MEM’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:434: error: ‘bptr’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:436: error: ‘BIO_f_base64’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:436: error: ‘BIO_new’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:437: error: ‘BIO_s_mem’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:438: error: ‘BIO_push’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:439: error: ‘BIO_write’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:440: error: ‘BIO_flush’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:441: error: ‘BIO_get_mem_ptr’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:444: error: ‘digestBuffer’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:446: error: ‘BIO_free_all’ was not declared in this scope
error: command 'gcc' failed with exit status 1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Supporting new versions of hadoop/cdh can be a non trivial task, since we have to patch the code that meanwhile can be changed. Recently we have worked for supporting hadoop 2.2.0, so cdh has been left aside for a while.
Now we have time for working on cdh 4.4/5, so they will be supported as soon as possible.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've been unable to build pydoop on my Centos6.2 machine via any method (pip, easy_install, or download and build). The version I am trying to install is 0.10.0. There are many build errors associated with include files, along the lines of:
In file included from /usr/include/python2.6/Python.h:42,
from /usr/include/boost/python/detail/wrap_python.hpp:142,
from /usr/include/boost/python/detail/prefix.hpp:13,
from /usr/include/boost/python/args.hpp:8,
from /usr/include/boost/python.hpp:11,
from src/hdfs_common.hpp:31,
from src/hdfs_fs.hpp:24,
from src/hdfs_fs.cpp:21:
/usr/include/stdlib.h:503: error: ‘size_t’ was not declared in this scope
/usr/include/stdlib.h:503: error: expected ‘,’ or ‘;’ before ‘throw’
/usr/include/stdlib.h:508: error: ‘size_t’ has not been declared
and on and on and on for pages . . .
Has anyone else encountered this and conquered it?
p.s. See blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/ for another user with similar build issues.
Hi, which version of Hadoop are you running?
Last edit: Simone Leo 2013-12-05
Cloudera CDH 4.4
Sorry, but at this moment Pydoop supports only CDH 4.2 and 4.3.
Interesting. We get the same errors on our CDH 4.3 cluster.
That'strange, did you install cdh from packages? Installation from tarball is not supported.
Why are only the older versions of CDH 4.X supported? Is there any way to get it to work with CDH 4.5? I have hadoop 2.0.0-cdh4.5.0 and am trying to install pydoop 0.11.1.
Ensuing errors are:
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:39:26: error: openssl/hmac.h: No such file or directory
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:40:28: error: openssl/buffer.h: No such file or directory
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc: In member function ‘std::string HadoopPipes::BinaryProtocol::createDigest(std::string&, std::string&)’:
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:423: error: ‘HMAC_CTX’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:423: error: expected ‘;’ before ‘ctx’
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:424: error: ‘EVP_MAX_MD_SIZE’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:425: error: ‘ctx’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:426: error: ‘EVP_sha1’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:426: error: ‘HMAC_Init’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:427: error: ‘HMAC_Update’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:429: error: ‘digest’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:429: error: ‘HMAC_Final’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:430: error: ‘HMAC_cleanup’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:433: error: ‘BIO’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:433: error: ‘bmem’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:433: error: ‘b64’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:434: error: ‘BUF_MEM’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:434: error: ‘bptr’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:436: error: ‘BIO_f_base64’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:436: error: ‘BIO_new’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:437: error: ‘BIO_s_mem’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:438: error: ‘BIO_push’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:439: error: ‘BIO_write’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:440: error: ‘BIO_flush’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:441: error: ‘BIO_get_mem_ptr’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:444: error: ‘digestBuffer’ was not declared in this scope
src/hadoop-2.0.0-cdh4.5.0.patched/pipes/impl/HadoopPipes.cc:446: error: ‘BIO_free_all’ was not declared in this scope
error: command 'gcc' failed with exit status 1
Supporting new versions of hadoop/cdh can be a non trivial task, since we have to patch the code that meanwhile can be changed. Recently we have worked for supporting hadoop 2.2.0, so cdh has been left aside for a while.
Now we have time for working on cdh 4.4/5, so they will be supported as soon as possible.
Your error seems related to openssl. See this thread for a possible solution:
http://sourceforge.net/p/pydoop/discussion/990018/thread/f189157f/