Impala2.7.0-cdh5.x.x安装部署
部署impala
impala安装选择rpm包方式进行，这是本次部署唯一一个主要主件采用rpm方式进行安装部署，这里主要原因是cloudera没有提供现成的tar包文件，而源码编译过程会出现各种未知原因，为了方便采用以下方式进行部署。
安装介质如下：$ lsbigtop-utils-0.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.noarch.rpmimpala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm impala-shell-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpmimpala-catalog-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm impala-state-store-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpmimpala-debuginfo-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm impala-udf-devel-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpmimpala-server-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpmsentry-1.5.1+cdh5.10.0+272-1.cdh5.10.0.p0.70.el7.noarch.rpm
[hadoop@db01 impala270]$ rpm -ivh impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm warning: impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEYerror: Failed dependencies: bigtop-utils >= 0.7 is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 hadoop is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 hadoop-hdfs is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 hadoop-yarn is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 hadoop-mapreduce is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 hbase is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 hive >= 0.12.0+cdh5.1.0 is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 zookeeper is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 hadoop-libhdfs is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 avro-libs is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 parquet is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 sentry >= 1.3.0+cdh5.1.0 is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 sentry is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 /lib/lsb/init-functions is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64 libhdfs.so.0.0.0()(64bit) is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64[hadoop@db01 impala270]$
1、安装bigtop和sentry
$sudo yum -y install redhat-lsb
sudo rpm -ivh bigtop-utils-0.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.noarch.rpm warning: bigtop-utils-0.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEYPreparing... ################################# [100%]Updating / installing... 1:bigtop-utils-0.7.0+cdh5.10.0+0-1.################################# [100%] $sudo rpm -ivh sentry-1.5.1+cdh5.10.0+272-1.cdh5.10.0.p0.70.el7.noarch.rpm --nodepswarning: sentry-1.5.1+cdh5.10.0+272-1.cdh5.10.0.p0.70.el7.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEYPreparing... ################################# [100%]Updating / installing... 1:sentry-1.5.1+cdh5.10.0+272-1.cdh5################################# [100%]
2、impalad安装
注：impalad需安装在所有datanode服务器上。
sudo rpm -ivh impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm --nodepswarning: impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEYPreparing... ################################# [100%]Updating / installing... 1:impala-2.7.0+cdh5.10.0+0-1.cdh5.1################################# [100%]
3、安装impala server
sudo rpm -ivh impala-server-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm warning: impala-server-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEYPreparing... ################################# [100%]Updating / installing... 1:impala-server-2.7.0+cdh5.10.0+0-1################################# [100%] --以上步骤在所有datanode节点安装 4、修改配置文件
sudo vim /etc/default/impala
IMPALA_CATALOG_SERVICE_HOST=db01IMPALA_STATE_STORE_HOST=db01
这两个ip是要装CATALOG和STATE_STORE的节点，CATALOG和STATE_STORE这两个服务必须和hive装在同一个节点在一个节点编辑后，复制到其他节点
sudo scp /etc/default/impala db02:/etc/default/impalasudo scp /etc/default/impala db03:/etc/default/impalasudo scp /etc/default/impala db04:/etc/default/impala
vim /etc/default/bigtop-utils
export JAVA_HOME=/opt/service/jdk1.7.0_67
scp bigtop-utils db02:/etc/default/bigtop-utilsscp bigtop-utils db03:/etc/default/bigtop-utilsscp bigtop-utils db04:/etc/default/bigtop-utils
vim hdfs-site.xml
dfs.datanode.hdfs-blocks-metadata.enabled true dfs.client.file-block-storage-locations.timeout.millis 10000
5、复制hive和hadoop的配置文件到impala
sudo scp /opt/cdh5/hive-1.1.0-cdh5.10.0/conf/hive-site.xml db01:/etc/impala/conf/sudo scp /opt/cdh5/hive-1.1.0-cdh5.10.0/conf/hive-site.xml db02:/etc/impala/conf/sudo scp /opt/cdh5/hive-1.1.0-cdh5.10.0/conf/hive-site.xml db03:/etc/impala/conf/sudo scp /opt/cdh5/hive-1.1.0-cdh5.10.0/conf/hive-site.xml db04:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/core-site.xml db01:/etc/impala/conf/sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/core-site.xml db02:/etc/impala/conf/sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/core-site.xml db03:/etc/impala/conf/sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/core-site.xml db04:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/hdfs-site.xml db01:/etc/impala/conf/sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/hdfs-site.xml db02:/etc/impala/conf/sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/hdfs-site.xml db03:/etc/impala/conf/sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/hdfs-site.xml db04:/etc/impala/conf/
6、在hive节点安装impala-state-store 和impala-catalog
$ sudo rpm -ivh impala-state-store-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm warning: impala-state-store-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEYPreparing... ################################# [100%]Updating / installing... 1:impala-state-store-2.7.0+cdh5.10.################################# [100%]$ sudo rpm -ivh impala-catalog-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm warning: impala-catalog-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEYPreparing... ################################# [100%]Updating / installing... 1:impala-catalog-2.7.0+cdh5.10.0+0-################################# [100%]
$ sudo cp /mnt/mysql-connector-java-5.1.22-bin.jar /var/lib/impala/
7、所有节点安装impala shell
sudo yum -y install python-setuptools sudo rpm -ivh impala-shell-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm 8、修改impala依赖jar包，通过软连接方式
sudo rm -rf /usr/lib/impala/lib/hadoop-annotations.jarsudo rm -rf /usr/lib/impala/lib/hadoop-auth.jarsudo rm -rf /usr/lib/impala/lib/hadoop-aws.jarsudo rm -rf /usr/lib/impala/lib/hadoop-common.jarsudo rm -rf /usr/lib/impala/lib/hadoop-hdfs.jarsudo rm -rf /usr/lib/impala/lib/hadoop-mapreduce-client-common.jarsudo rm -rf /usr/lib/impala/lib/hadoop-mapreduce-client-core.jarsudo rm -rf /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jarsudo rm -rf /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jarsudo rm -rf /usr/lib/impala/lib/hadoop-yarn-api.jarsudo rm -rf /usr/lib/impala/lib/hadoop-yarn-client.jarsudo rm -rf /usr/lib/impala/lib/hadoop-yarn-common.jarsudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jarsudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-common.jarsudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jarsudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jarsudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jarsudo rm -rf /usr/lib/impala/lib/hbase-annotations.jarsudo rm -rf /usr/lib/impala/lib/hbase-client.jarsudo rm -rf /usr/lib/impala/lib/hbase-common.jarsudo rm -rf /usr/lib/impala/lib/hbase-protocol.jarsudo rm -rf /usr/lib/impala/lib/hive-ant.jarsudo rm -rf /usr/lib/impala/lib/hive-beeline.jarsudo rm -rf /usr/lib/impala/lib/hive-common.jarsudo rm -rf /usr/lib/impala/lib/hive-exec.jarsudo rm -rf /usr/lib/impala/lib/hive-hbase-handler.jarsudo rm -rf /usr/lib/impala/lib/hive-metastore.jarsudo rm -rf /usr/lib/impala/lib/hive-serde.jarsudo rm -rf /usr/lib/impala/lib/hive-service.jarsudo rm -rf /usr/lib/impala/lib/hive-shims-common.jarsudo rm -rf /usr/lib/impala/lib/hive-shims.jarsudo rm -rf /usr/lib/impala/lib/hive-shims-scheduler.jarsudo rm -rf /usr/lib/impala/lib/zookeeper.jar sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-annotations.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-auth.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce1/lib/hadoop-aws-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-aws.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/common/hadoop-common-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-common.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-hdfs.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-core.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-api-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-api.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-client-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-client.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-common-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-common.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-common.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jarsudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jarsudo ln -s /opt/cdh5/hbase-1.2.0-cdh5.10.0/lib/hbase-annotations-1.2.0-cdh5.10.0.jar /usr/lib/impala/lib/hbase-annotations.jarsudo ln -s /opt/cdh5/hbase-1.2.0-cdh5.10.0/lib/hbase-client-1.2.0-cdh5.10.0.jar /usr/lib/impala/lib/hbase-client.jarsudo ln -s /opt/cdh5/hbase-1.2.0-cdh5.10.0/lib/hbase-common-1.2.0-cdh5.10.0.jar /usr/lib/impala/lib/hbase-common.jarsudo ln -s /opt/cdh5/hbase-1.2.0-cdh5.10.0/lib/hbase-protocol-1.2.0-cdh5.10.0.jar /usr/lib/impala/lib/hbase-protocol.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-ant-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-ant.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-beeline-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-beeline.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-common-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-common.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-exec-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-exec.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-hbase-handler-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-hbase-handler.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-metastore-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-metastore.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-serde-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-serde.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-service-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-service.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-shims-common-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-shims-common.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-shims-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-shims.jarsudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-shims-scheduler-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-shims-scheduler.jarsudo ln -s /opt/cdh5/zookeeper-3.4.5-cdh5.10.0/zookeeper-3.4.5-cdh5.10.0.jar /usr/lib/impala/lib/zookeeper.jar
rm -rf /usr/lib/impala/lib/libhadoop.sorm -rf /usr/lib/impala/lib/libhadoop.so.1.0.0rm -rf /usr/lib/impala/lib/libhdfs.sorm -rf /usr/lib/impala/lib/libhdfs.so.0.0.0
ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhadoop.so /usr/lib/impala/lib/libhadoop.soln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhadoop.so.1.0.0 /usr/lib/impala/lib/libhadoop.so.1.0.0ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhdfs.so /usr/lib/impala/lib/libhdfs.soln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhdfs.so.0.0.0 /usr/lib/impala/lib/libhdfs.so.0.0.0
9、启动impala
#安装了hive的节点启动state-store和catalog服务
service impala-state-store startservice impala-catalog start
#所有datanode节点启动impalad
service impala-server start
版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。
暂时没有评论，来抢沙发吧~