躬行系列-hadoop集群部署
文章目录
Java安装
[root@localhost jdk-11.0.15]# cd /etc/profile.d/
[root@localhost profile.d]# vim my_env.sh
[root@localhost profile.d]# source /etc/profile机器初始配置
因为我这是虚拟机,所以要配置固定ip。如果你也和我一样是自己测试,一定注意netmask和gateway要更换成自己的。
vim /etc/sysconfig/network-scripts/ifcfg-enp0s3
-------------------------------------
IPADDR="192.168.0.114"
NETMASK="255.255.255.0"
GATEWAY="192.168.0.1"
DNS1="114.114.114.114"hostname
hostnamectl set-hostname hadoop-1
cat >> /etc/hosts << EOF
192.168.0.114 hadoop-1
192.168.0.115 hadoop-2
192.168.0.116 hadoop-3
EOF集群规划
| ip | name | 组件 |
|---|---|---|
| 192.168.0.114 | hadoop-1 | namenode datanode journalnode |
| 192.168.0.115 | hadoop-2 | namenode datanode journalnode |
| 192.168.0.116 | hadoop-3 | namenode datanode journalnode |
配置
上传解压等忽略
两个核心配置文件:core-site.xml hdfs-site.xml
位置:/usr/hadoop/hadoop-3.3.3/etc/hadoop
环境变量
配置自己的环境变量地址,无需配置在原有的系统profile中/etc/profile.d/my_env.sh
启动的话只需要source /etc/profile.d/my_env.sh
export HADOOP_HOME=/usr/local/hadoop/hadoop-3.3.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
source /etc/profile
hadoop version常用命令
启动journalnodehdfs --daemon start journalnode
启动namenodehdfs namenode -formathdfs --daemon start namenodehdfs namenode -bootstrapStandby
zookeeper启动./zkServer.sh start
zkfc初始化hdfs zkfc -formatZK
集群一键启停start-dfs.shstop-dfs.shstart-yarn.shstop-yarn.sh
获取节点状态hdfs haadmin -getServiceState nn2
手动切换节点状态为activehdfs haadmin -transitionToActive --forcemanual nn2
hadoop手动上传文件hadoop fs -put README.md /
获取yarn节点状态yarn rmadmin -getServiceState rm1
查询zookeeper集群状态./zkServer.sh status
启停history server./mapred --daemon start historyserver
web控制台
hadoop:http://192.168.0.116:9870/
yarn:http://hadoop-1:8088/
history server:http://hadoop-1:19888/jobhistory
Hadoop性能测试
写测试hadoop jar /usr/local/hadoop/hadoop-3.3.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.3-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 128MB
读测试hadoop jar /usr/local/hadoop/hadoop-3.3.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.3-tests.jar TestDFSIO -read -nrFiles 5 -fileSize 128MB
mapreduce测试
至少要10台服务器+1T*128G才能跑
启动kafka
bin/kafka-server-start.sh -daemon /usr/local/kafka/kafka_2.13-3.2.1/config/server.properties
评论
登录后参与技术讨论,分享你的经验与见解。
登录 / 注册暂无评论