暂无说说

hadoop集群搭建

hadoop jiajun 11个月前 (12-04) 202次浏览 0个评论 扫描二维码

一、环境说明

节点:s100~s102

192.168.163.100 s100

192.168.163.101 s101

192.168.163.102 s102

操作系统:centos7

hadoop 版本:hadoop2.7.2

jdk 版本:1.8

二、安装过程

1、修改 hosts

sudo vi /etc/hosts

添加

192.168.163.100 s100
192.168.163.101 s101
192.168.163.102 s102
192.168.163.100 localhost   #如果没个,pyspark 启动报 Name or service not known

分发 

rsync /etc/hosts root@s101:/etc/
rsync /etc/hosts root@s102:/etc/

2、配置 ssh 免密码登陆

ssh-keygen -t rsa
ssh-copy-id s100
ssh-copy-id s101
ssh-copy-id s102

3、安装 jdk、hadoop

解压

sudo mkdir /soft
sudo chmod hadoop:hadoop /soft
tar -zxvf jdk-8u151-linux-x64.tar.gz -C /soft/
ln -s /soft/jdk1.8.0_151/ /soft/jdk
tar -zxvf hadoop-2.7.2.tar.gz -C /soft/
ln -s /soft/hadoop-2.7.2/ /soft/hadoop

修改环境变量

sudo vi /etc/profile

添加以下内容

export JAVA_HOME=/soft/jdk
export HADOOP_HOME=/soft/hadoop
export LD_LIBRARY_PATH=/soft/hadoop/lib/native
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin


修改 hadoop 配置

cd /soft/hadoop/etc/hadoop

vi core-site.xml

<configuration>
  <property>
     <name>fs.defaultFS</name>
     <value>hdfs://s100:9000</value>
  </property>
  <property>
     <name>hadoop.tmp.dir</name>
     <value>/soft/hadoop/data</value>
  </property>
</configuration>

vi hdfs-site.xml

<configuration>
  <property>
      <name>dfs.replication</name>
      <value>3</value>
  </property>
  <property>
      <name>dfs.permissions</name>
      <value>false</value>
  </property>
</configuration>

vi yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
  <property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle</value>
  </property>
<!-- 出现容器被 kill 试着改这个,默认 2.1 -->
  <property>
     <name>yarn.nodemanager.vmem-pmem-ratio</name>
     <value>3</value>
  </property>
</configuration>

cp mapred-site.xml.template mapred-site.xml

vi mapred-site.xml

<configuration>
  <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
  </property>
</configuration>

vi slaves

s100
s101
s102

vi hadoop-env.sh

export JAVA_HOME=/soft/jdk

分发

rsync /etc/profile root@s101:/etc/
rsync /etc/profile root@s102:/etc/
rsync -lr /soft/hadoop* hadoop@s101:/soft/
rsync -lr /soft/hadoop* hadoop@s102:/soft/
rsync -lr /soft/jdk* hadoop@s101:/soft/
rsync -lr /soft/jdk* hadoop@s102:/soft/

在三个节点上刷新环境变量

source /etc/profile

格式化 hadoop

hdfs namenode -format

4、启动 hadoop 集群

start-all.sh

s100

jps
10210 Jps
9523 DataNode
9672 SecondaryNameNode
9816 ResourceManager
9404 NameNode
10093 NodeManager

s101

jps
9348 Jps
9127 DataNode
9241 NodeManager

s102

9338 Jps
9116 DataNode
9231 NodeManager

访问 webui

http://192.168.163.100:50070

http://192.168.163.100:8088

喜欢 (0)
发表我的评论
取消评论

表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址