大数据各组件搭建,版本冲突问题解决方案
文章目录
最近由于搭建hadoop、flink、kafka的环境,需要做一整套大数据的方案。
分别取相应的官网上查看几个组件的最新情况。
hadoop选用CDH,目前最新版本为CDH6.0.0 对应的hbase、hive、spark等组件的版本如下:
| 组件 | 版本 | 发行版 | CDH 版本 |
|---|---|---|---|
| Supervisord | 3.0 | 不可用 | 不适用 |
| Cloudera Manager Agent | 6.0.0 | 530873.el7 | 不适用 |
| Cloudera Manager Management Daemon | 6.0.0 | 530873.el7 | 不适用 |
| Flume NG | 1.8.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Hadoop | 3.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| HDFS | 3.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| HttpFS | 3.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| hadoop-kms | 3.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| MapReduce | 2 3.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| YARN | 3.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| HBase | 2.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Lily HBase Indexer | 1.5+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Hive | 2.1.1+cdh6.0.0 | 537114 | CDH 6.0.0 |
| HCatalog | 2.1.1+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Hue | 3.9.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Impala | 3.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Java 8 | java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) | 不可用 | 不适用 |
| Kafka | 1.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Kite(仅限 CDH 5 ) | 1.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| kudu | 1.6.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Oozie | 5.0.0-beta1+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Parquet | 1.9.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Pig | 0.17.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| sentry | 2.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Solr | 7.0.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| spark | 2.2.0+cdh6.0.0 | 537114 | CDH 6.0.0 |
| Sqoop | 1.4.7+cdh6.0.0 | 537114 | CDH 6.0.0 |
| ZooKeeper | 3.4.5+cdh6.0.0 | 537114 | CDH 6.0.0 |
flink最新的release版本为1.6
- Flink 1.6.0 - 2018-08-08 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.5.3 - 2018-08-21 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.5.2 - 2018-07-31 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.5.1 - 2018-07-12 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.5.0 - 2018-05-25 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.4.2 - 2018-03-08 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.4.1 - 2018-02-15 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.4.0 - 2017-11-29 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.3.3 - 2018-03-15 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.3.2 - 2017-08-05 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.3.1 - 2017-06-23 (Source, Binaries, Docs, Javadocs, ScalaDocs)
- Flink 1.3.0 - 2017-06-01 (Source, Binaries, Docs, Javadocs, ScalaDocs)
kafka最新版本2.0.0
- Released July 30, 2018
由于kafka2.0有很多新的特性比较吸引我。果断安装了最新版。
然后问题来了·······
flink1.6 release 中自带的example,运行了一下kafka的example,成功连上kafka,但是发现发送到kafka的数据,flink根本无法消费到,并且运行中出现无法识别配置的警告。
13:50:15,604 WARN org.apache.kafka.clients.producer.ProducerConfig - The configuration 'output-topic' was supplied but isn't a known config.
13:50:15,604 WARN org.apache.kafka.clients.producer.ProducerConfig - The configuration 'zookeeper.connect' was supplied but isn't a known config.
13:50:15,605 WARN org.apache.kafka.clients.producer.ProducerConfig - The configuration 'input-topic' was supplied but isn't a known config.
一想可能还是各个组件的版本问题,赶紧去查了一下当前example中kafka-client的版本,一查果然:
flink通过flink-connector-kafka-0.10_${scala.binary.version}的包去kafka读取数据,
而flink-connector-kafka中kafka-clients版本为:
<properties>
<kafka.version>0.10.2.1</kafka.version>
</properties>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.version}</version>
</dependency>
而我装的kafka确是2.0.0,整整差了两个大版本,也就难怪会出问题了。
最后查看了一下目前flink-connector-kafka支持的最新版本为0.11,所以考虑单独安装0.11的kafka,flink1.6以及CDH6.0.0。
flink1.6也可以在CDH6.0.0上的yarn运行。
但目前看趟坑应该远未结束。祝我好运吧。
| 组件 | 版本 | 发行版 | CDH 版本 |
|---|---|---|---|
| Kafka | 0.10.2+kafka2.2.0 | 1.2.2.0.p0.92 | 不适用 |
评论
登录后参与技术讨论,分享你的经验与见解。
登录 / 注册暂无评论