Spark HA集群搭建
文章目录
本教程基于Hadoop HA集群搭建。
版本介绍
| software | version |
|---|---|
| OS | CentOS-7-x86_64-DVD-1810.iso |
| Hadoop | hadoop-2.8.4 |
| Zookeeper | zookeeper-3.4.10 |
| Spark | spark-2.4.3 |
集群角色分配
| node | actor |
|---|---|
| master1 | NameNode、DFSZKFailoverController(zkfc)、ResourceManager、Master |
| master2 | NameNode、DFSZKFailoverController(zkfc)、ResourceManager、Master |
| node1 | DataNode、NodeManager、JournalNode、QuorumPeerMain、Worker |
| node2 | DataNode、NodeManager、JournalNode、QuorumPeerMain、Worker |
| node3 | DataNode、NodeManager、JournalNode、QuorumPeerMain、Worker |
配置Spark [all]
解压重命名
tar -xvf spark-2.4.3-bin-hadoop2.7.tgz -C /opt/spark/
mv spark-2.4.3-bin-hadoop2.7 spark-2.4.3
mv sbin/start-all.sh sbin/start-spark-all.sh
mv sbin/stop-all.sh sbin/stop-spark-all.sh
配置环境变量
vi ~/.bash_profile
export SPARK_HOME=/opt/spark/spark-2.4.3
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
source ~/.bash_profile
配置spark运行环境
vi /opt/spark/spark-2.4.3/conf/spark-env.sh
export JAVA_HOME=/opt/env/jdk1.8.0_121
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=1g
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=node1:2181,node2:2181,node3:2181 -Dspark.dep
loy.zookeeper.dir=/spark/ha"
配置spark slaves
vi /opt/spark/spark-2.4.3/conf/slaves
node1
node2
node3
启动集群[master1 master2]
[hadoop@master1 spark-2.4.3]$ sbin/start-spark-all.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark/spark-2.4.3/logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-master1.out
node2: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/spark-2.4.3/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-node2.out
node3: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/spark-2.4.3/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-node3.out
node1: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/spark-2.4.3/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-node1.out
[hadoop@master2 spark-2.4.3]$ sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark/spark-2.4.3/logs/spark-hadoop-org.apache.spark.deploy.master.Master-2-master2.out
验证[all]
[hadoop@master1 hadoop-2.8.4]$ jps
3808 Master
4390 ResourceManager
4281 DFSZKFailoverController
4474 Jps
3981 NameNode
[hadoop@node3 sbin]$ jps
3377 QuorumPeerMain
3793 NodeManager
3491 Worker
3589 DataNode
3673 JournalNode
3935 Jps

评论
登录后参与技术讨论,分享你的经验与见解。
登录 / 注册暂无评论