1. 修改配置文件
- {hadoop_home}/etc/hadoop/hadoop-env.sh
修改jdk路径
- {hadoop_home}/etc/hadoop/hdfs-site.xml
dfs.namenode.secondary.http-address HADOOP01:50090 dfs.replication 3 dfs.permissions false dfs.namenode.http-address HADOOP01:50070 dfs.namenode.name.dir /home/bigdata/hadata/dfs/name dfs.datanode.data.dir file:///home/bigdata/hadoop/data/dfs/dn dfs.nameservices yulang dfs.ha.namenodes.yulang nn1,nn2 dfs.namenode.rpc-address.yulang.nn1 HADOOP01:8020 dfs.namenode.rpc-address.yulang.nn2 HADOOP02:8020 dfs.namenode.http-address.yulang.nn1 HADOOP01:50070 dfs.namenode.http-address.yulang.nn2 HADOOP02:50070 dfs.namenode.shared.edits.dir qjournal://HADOOP01:8485;HADOOP02:8485;HADOOP03/yulang dfs.ha.automatic-failover.enabled true dfs.client.failover.proxy.provider.yulang org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methods sshfence dfs.ha.fencing.ssh.private-key-files /home/bigdata/.ssh/id_rsa dfs.ha.fencing.ssh.connect-timeout 30000 dfs.journalnode.edits.dir /home/bigdata/hadata/journal
- yarn-site.xml
yarn.resourcemanager.resource-tracker.address HADOOP01:8031 yarn.resourcemanager.ha.enabled true yarn.resourcemanager.cluster-id yulang_yarn yarn.resourcemanager.ha.rm-ids rm1,rm2 yarn.resourcemanager.hostname.rm1 HADOOP01 yarn.resourcemanager.hostname.rm2 HADOOP02 yarn.resourcemanager.webapp.address.rm1 HADOOP01:8088 yarn.resourcemanager.webapp.address.rm2 HADOOP02:8088 yarn.resourcemanager.zk-address HADOOP01:2181,HADOOP02:2181,HADOOP03:2181 yarn.nodemanager.resource.memory-mb 1536 yarn.nodemanager.aux-services mapreduce_shuffle
- core-site.xml
hadoop.tmp.dir /home/bigdata/hadoop/data fs.defaultFS hdfs://yulang fs.trash.interval 60 ha.zookeeper.quorum HADOOP01:2181,HADOOP02:2181,HADOOP03:2181
2. 将配置信息通过ssh免密协议传输到其他的服务器上
scp -r /home/bigdata/hadoop-2.7.2/etc/hadoop/ HADOOP02:/home/bigdata/hadoop-2.7.2/etc/
scp -r /home/bigdata/hadoop-2.7.2/etc/hadoop/ HADOOP03:/home/bigdata/hadoop-2.7.2/etc/
3. 启动高可用集群
- hadoop-deamons.sh start journalnode
启动journalnode集群
- hdfs namenode -format
选择一台namenode进行格式化并启动
- hdfs namenode -bootstraoStandby
在另一台namenode服务器进行拉取元数据操作(或者手动copy数据过来)
- hdfs zkfc -formatZK
格式化zkfc
- start-dfs.sh //并检查节点都是否启动，若没有，则单独启动(hadoop-deamon.sh start xxx)
启动hdfs模块
//namenode服务器主要的节点有：naemnode、datanode、journalnode、DFSZKFailoverController、QuorumPeerMain//子节点服务器主要的节点有：JournalNode、datanode、QuorumPeerMain
- start-yarn.sh //另一台namenode服务器需要单独启动resourceManager节点
启动yarn模块