Linux安装Kafka和简单使用

1、下载 https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.0.0/kafka_2.11-2.0.0.tgz

2、解压 tar -xvf kafka_2.11-2.0.0.tgz

3、设置环境变量

  • vim /etc/profile 或者 vim ~/.bashrc

    #Kafka Environment export KAFKA_HOME=$USER_HOME/software/kafka_2.11-2.0.0 source /etc/profile 或者 source ~/.bashrc

4、修改配置文件

  • vim $KAFKA_HOME/config/server.properties

    #listeners=PLAINTEXT://:9092

改成

listeners=PLAINTEXT://:9092


log.dirs=/tmp/kafka-logs

改成

log.dirs=/home/hadoop/tmp/kafka-logs

使用非自带的Zookeeper,不用改动下面的配置

  • vim $KAFKA_HOME/config/zookeeper.properties

    dataDir=/tmp/zookeeper

改成

dataDir=/home/hadoop/tmp/zookeeper
dataLogDir=/home/hadoop/logs/zookeeper

5、原始启动

  • 启动Zookeeper

    nohup ~/software/kafka_2.11-2.0.0/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties &

  • 启动Kafka

    nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

6、起停脚本(需提前安装zookeeper)

~/software/kafka_2.11-2.0.0/bin/start-kafka.sh
~/software/kafka_2.11-2.0.0/bin/stop-kafka.sh
  • vim ~/software/kafka_2.11-2.0.0/bin/start-kafka.sh

    #Start Zookeeper nohup ~/software/kafka_2.11-2.0.0/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties & #Start Kafka nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

  • vim ~/software/kafka_2.11-2.0.0/bin/stop-kafka.sh

    #Stop Kafka nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-stop.sh $KAFKA_HOME/config/server.properties & #Stop Zookeeper nohup ~/software/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh $KAFKA_HOME/config/zookeeper.properties &

使用非自带的Zookeeper,不用改动下面的配置

  • vim ~/software/kafka_2.11-2.0.0/bin/start-kafka.sh

    #Start Kafka #nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties >$USER_HOME/logs/kafka/kafka_start.log & nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties >$USER_HOME/logs/kafka/kafka_start.log 2>&1 &

  • vim ~/software/kafka_2.11-2.0.0/bin/stop-kafka.sh

    #Stop Kafka #nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-stop.sh $KAFKA_HOME/config/server.properties >$USER_HOME/logs/kafka/kafka_start.log & nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-stop.sh $KAFKA_HOME/config/server.properties >$USER_HOME/logs/kafka/kafka_start.log 2>&1 &

7、添加执行权限

chmod +x ~/software/kafka_2.11-2.0.0/bin/start-kafka.sh ~/software/kafka_2.11-2.0.0/bin/stop-kafka.sh

8、Kafka常用命令 创建Topic

~/software/kafka_2.11-2.0.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看创建的Topic

~/software/kafka_2.11-2.0.0/bin/kafka-topics.sh -list -zookeeper localhost:2181

启动生产者

~/software/kafka_2.11-2.0.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

在另外一个终端启动消费者

~/software/kafka_2.11-2.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

9、FAQ [2019-04-23 16:07:30,396] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists rm -rf /home/hadoop/tmp/kafka-logs 清理日志信息

评论

暂无评论