Hadoop: Python client driver for HiveServer2 fails

Question

I am trying to install a Python client driver for HiveServer2: https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-PythonClientDriver

Installations says that: "A Python client driver for HiveServer2 is available at https://github.com/BradRuderman/pyhs2 It includes all the required packages such as SASL and Thrift wrappers".

Yet, running pip install pyhs2 fails on SASL compilation (see below). I have Hadoop 2.2.0 installed and working on localhost. Please help, to install Python client.

[root@localhost /]# pip install pyhs2
Requirement already satisfied (use --upgrade to upgrade): pyhs2 in /usr/lib/python2.6/site-packages
Downloading/unpacking sasl (from pyhs2)
  Downloading sasl-0.1.3.tar.gz
  Running setup.py (path:/tmp/pip_build_root/sasl/setup.py) egg_info for package sasl
Downloading/unpacking thrift (from pyhs2)
  Downloading thrift-0.9.1.tar.gz
  Running setup.py (path:/tmp/pip_build_root/thrift/setup.py) egg_info for package thrift
Installing collected packages: sasl, thrift
  Running setup.py install for sasl
    building '_saslwrapper' extension
                                                                                                                                                                                                                                                                                                                                                                    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.6 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.6/sasl/saslwrapper.o
sasl/saslwrapper.cpp:21:23: error: sasl/sasl.h: No such file or directory
sasl/saslwrapper.cpp:51: error: ‘sasl_interact_t’ has not been declared
sasl/saslwrapper.cpp:54: error: ‘sasl_conn_t’ has not been declared
sasl/saslwrapper.cpp:54: error: ‘sasl_secret_t’ has not been declared
sasl/saslwrapper.cpp:57: error: ISO C++ forbids declaration of ‘sasl_conn_t’ with no type
sasl/saslwrapper.cpp:57: error: expected ‘;’ before ‘*’ token
sasl/saslwrapper.cpp:58: error: ‘sasl_callback_t’ does not name a type
sasl/saslwrapper.cpp:71: error: ISO C++ forbids declaration of ‘sasl_secret_t’ with no type
sasl/saslwrapper.cpp:71: error: expected ‘;’ before ‘*’ token
sasl/saslwrapper.cpp: In constructor ‘saslwrapper::ClientImpl::ClientImpl()’:
sasl/saslwrapper.cpp:35: error: class ‘saslwrapper::ClientImpl’ does not have any field named ‘conn’
sasl/saslwrapper.cpp:35: error: class ‘saslwrapper::ClientImpl’ does not have any field named ‘secret’
sasl/saslwrapper.cpp: In destructor ‘saslwrapper::ClientImpl::~ClientImpl()’:
sasl/saslwrapper.cpp:36: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:36: error: ‘sasl_dispose’ was not declared in this scope
sasl/saslwrapper.cpp:36: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘void saslwrapper::ClientImpl::lastCallback()’:
sasl/saslwrapper.cpp:49: error: ‘SASL_CB_LIST_END’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::init()’:
sasl/saslwrapper.cpp:83: error: ‘sasl_client_init’ was not declared in this scope
sasl/saslwrapper.cpp:84: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:85: error: ‘sasl_errstring’ was not declared in this scope
sasl/saslwrapper.cpp:91: error: ‘SASL_CB_GETREALM’ was not declared in this scope
sasl/saslwrapper.cpp:93: error: ‘SASL_CB_USER’ was not declared in this scope
sasl/saslwrapper.cpp:94: error: ‘SASL_CB_AUTHNAME’ was not declared in this scope
sasl/saslwrapper.cpp:97: error: ‘SASL_CB_PASS’ was not declared in this scope
sasl/saslwrapper.cpp:99: error: ‘SASL_CB_PASS’ was not declared in this scope
sasl/saslwrapper.cpp:107: error: ‘SASL_NEED_PROXY’ was not declared in this scope
sasl/saslwrapper.cpp:109: error: ‘callbacks’ was not declared in this scope
sasl/saslwrapper.cpp:109: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:109: error: ‘sasl_client_new’ was not declared in this scope
sasl/saslwrapper.cpp:110: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:111: error: ‘sasl_errstring’ was not declared in this scope
sasl/saslwrapper.cpp:115: error: ‘sasl_security_properties_t’ was not declared in this scope
sasl/saslwrapper.cpp:115: error: expected ‘;’ before ‘secprops’
sasl/saslwrapper.cpp:117: error: ‘secprops’ was not declared in this scope
sasl/saslwrapper.cpp:124: error: ‘SASL_SEC_PROPS’ was not declared in this scope
sasl/saslwrapper.cpp:124: error: ‘sasl_setprop’ was not declared in this scope
sasl/saslwrapper.cpp:125: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:127: error: ‘sasl_dispose’ was not declared in this scope
sasl/saslwrapper.cpp:133: error: ‘SASL_AUTH_EXTERNAL’ was not declared in this scope
sasl/saslwrapper.cpp:134: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:136: error: ‘sasl_dispose’ was not declared in this scope
sasl/saslwrapper.cpp:141: error: ‘SASL_SSF_EXTERNAL’ was not declared in this scope
sasl/saslwrapper.cpp:142: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:144: error: ‘sasl_dispose’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::setAttr(const std::string&, const std::string&)’:
sasl/saslwrapper.cpp:163: error: ‘secret’ was not declared in this scope
sasl/saslwrapper.cpp:164: error: ‘sasl_secret_t’ was not declared in this scope
sasl/saslwrapper.cpp:164: error: expected primary-expression before ‘)’ token
sasl/saslwrapper.cpp:164: error: expected ‘;’ before ‘malloc’
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::start(const std::string&, saslwrapper::output_string&, saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:199: error: ‘sasl_interact_t’ was not declared in this scope
sasl/saslwrapper.cpp:199: error: ‘prompt’ was not declared in this scope
sasl/saslwrapper.cpp:205: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:205: error: ‘sasl_client_start’ was not declared in this scope
sasl/saslwrapper.cpp:206: error: ‘SASL_INTERACT’ was not declared in this scope
sasl/saslwrapper.cpp:208: error: ‘SASL_INTERACT’ was not declared in this scope
sasl/saslwrapper.cpp:209: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:209: error: ‘SASL_CONTINUE’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::step(const std::string&, saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:222: error: ‘sasl_interact_t’ was not declared in this scope
sasl/saslwrapper.cpp:222: error: ‘prompt’ was not declared in this scope
sasl/saslwrapper.cpp:227: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:227: error: ‘sasl_client_step’ was not declared in this scope
sasl/saslwrapper.cpp:228: error: ‘SASL_INTERACT’ was not declared in this scope
sasl/saslwrapper.cpp:230: error: ‘SASL_INTERACT’ was not declared in this scope
sasl/saslwrapper.cpp:231: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:231: error: ‘SASL_CONTINUE’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::encode(const std::string&, saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:244: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:244: error: ‘sasl_encode’ was not declared in this scope
sasl/saslwrapper.cpp:245: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::decode(const std::string&, saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:265: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:265: error: ‘sasl_decode’ was not declared in this scope
sasl/saslwrapper.cpp:266: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::getUserId(saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:282: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:282: error: ‘SASL_USERNAME’ was not declared in this scope
sasl/saslwrapper.cpp:282: error: ‘sasl_getprop’ was not declared in this scope
sasl/saslwrapper.cpp:283: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::getSSF(saslwrapper::output_int*)’:
sasl/saslwrapper.cpp:294: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:294: error: ‘SASL_SSF’ was not declared in this scope
sasl/saslwrapper.cpp:294: error: ‘sasl_getprop’ was not declared in this scope
sasl/saslwrapper.cpp:295: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘void saslwrapper::ClientImpl::addCallback(long unsigned int, void*)’:
sasl/saslwrapper.cpp:311: error: ‘callbacks’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘void saslwrapper::ClientImpl::setError(const std::string&, int, const std::string&, const std::string&)’:
sasl/saslwrapper.cpp:323: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:324: error: ‘sasl_errdetail’ was not declared in this scope
sasl/saslwrapper.cpp:326: error: ‘sasl_errstring’ was not declared in this scope
sasl/saslwrapper.cpp: At global scope:
sasl/saslwrapper.cpp:338: error: variable or field ‘interact’ declared void
sasl/saslwrapper.cpp:338: error: ‘sasl_interact_t’ was not declared in this scope
sasl/saslwrapper.cpp:338: error: ‘prompt’ was not declared in this scope
error: command 'gcc' failed with exit status 1

Answer 1

If you have an error in Windows when installing "sasl" with

pip install sasl

you can install pre-compiled packet.

Download sasl packet for python36 in x64 from link: https://www.lfd.uci.edu/~gohlke/pythonlibs/#sasl

Then run pip install sasl-0.2.1-cp36-cp36m-win_amd64.whl

Answer 2

pyhs2 is great, it packages several needed python packages together for a dbapi, among them sasl. Note if you pip install sasl you'll get the same error.

sasl (the python package) depends on libsasl2-dev (on a Debian/Ubuntu machine). The error you're seeing is that the compiler can't find the libraries that sasl is supposed to wrap.

If you sudo apt-get install libsasl2-dev and rerun, it will likely be fixed.

edit: Based on your comment, here's a robust way of getting this to work:

curl -O -L ftp://ftp.cyrusimap.org/cyrus-sasl/cyrus-sasl-2.1.26.tar.gz
tar xzf cyrus-sasl-2.1.26.tar.gz
cd cyrus-sasl-2.1.26
./configure && make install

Answer 3

Pyhs2 has the following dependencies:

1. gcc-c++

2. python-devel.x86_64

3. cyrus-sasl-devel.x86_64

So, just run this

sudo yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64

sudo pip install pyhs2