如何將Hadoop部署在低廉的硬件上
一、概述
Hadoop 是 Apache 軟件基金會下一個(gè)開源分布式計(jì)算平臺，以 HDFS（Hadoop Distributed File System）、MapReduce（Hadoop2.0 加入了 YARN，Yarn 是資源調(diào)度框架，能夠細(xì)粒度的管理和調(diào)度任務(wù)，還能夠支持其他的計(jì)算框架，比如 spark）為核心的 Hadoop 為用戶提供了系統(tǒng)底層細(xì)節(jié)透明的分布式基礎(chǔ)架構(gòu)。hdfs 的高容錯(cuò)性、高伸縮性、高效性等優(yōu)點(diǎn)讓用戶可以將 Hadoop 部署在低廉的硬件上，形成分布式系統(tǒng)。
HDFS 
YARN
二、開始部署
1）添加源
地址：
https://artifacthub.io/packages/helm/apache-hadoop-helm/hadoop
helmrepoaddapache-hadoop-helmhttps://pfisterer.github.io/apache-hadoop-helm/ helmpullapache-hadoop-helm/hadoop--version1.2.0 tar-xfhadoop-1.2.0.tgz
2）構(gòu)建鏡像 Dockerfile
FROMmyharbor.com/bigdata/centos:7.9.2009 RUNrm-f/etc/localtime&&ln-sv/usr/share/zoneinfo/Asia/Shanghai/etc/localtime&&echo"Asia/Shanghai">/etc/timezone RUNexportLANG=zh_CN.UTF-8 #創(chuàng)建用戶和用戶組，跟yaml編排里的spec.template.spec.containers.securityContext.runAsUser:9999 RUNgroupadd--system--gid=9999admin&&useradd--system--home-dir/home/admin--uid=9999--gid=adminadmin #安裝sudo RUNyum-yinstallsudo;chmod640/etc/sudoers #給admin添加sudo權(quán)限 RUNecho"adminALL=(ALL)NOPASSWD:ALL">>/etc/sudoers RUNyum-yinstallinstallnet-toolstelnetwget RUNmkdir/opt/apache/ ADDjdk-8u212-linux-x64.tar.gz/opt/apache/ ENVJAVA_HOME=/opt/apache/jdk1.8.0_212 ENVPATH=$JAVA_HOME/bin:$PATH ENVHADOOP_VERSION3.3.2 ENVHADOOP_HOME=/opt/apache/hadoop ENVHADOOP_COMMON_HOME=${HADOOP_HOME} HADOOP_HDFS_HOME=${HADOOP_HOME} HADOOP_MAPRED_HOME=${HADOOP_HOME} HADOOP_YARN_HOME=${HADOOP_HOME} HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop PATH=${PATH}:${HADOOP_HOME}/bin #RUNcurl--silent--output/tmp/hadoop.tgzhttps://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz&&tar--directory/opt/apache-xzf/tmp/hadoop.tgz&&rm/tmp/hadoop.tgz ADDhadoop-${HADOOP_VERSION}.tar.gz/opt/apache RUNln-s/opt/apache/hadoop-${HADOOP_VERSION}${HADOOP_HOME} RUNchown-Radmin:admin/opt/apache WORKDIR$HADOOP_HOME #Hdfsports EXPOSE500105002050070500755009080209000 #Mapredports EXPOSE19888 #Yarnports EXPOSE8030803180328033804080428088 #Otherports EXPOSE497072122
開始構(gòu)建鏡像
dockerbuild-tmyharbor.com/bigdata/hadoop:3.3.2.--no-cache ###參數(shù)解釋 #-t：指定鏡像名稱 # . ：當(dāng)前目錄Dockerfile #-f：指定Dockerfile路徑 #--no-cache：不緩存
推送到鏡像倉庫
dockerpushmyharbor.com/bigdata/hadoop:3.3.2
調(diào)整目錄結(jié)構(gòu)
mkdirhadoop/templates/hdfshadoop/templates/yarn mvhadoop/templates/hdfs-*hadoop/templates/hdfs/ mvhadoop/templates/yarn-*hadoop/templates/yarn/
2）修改配置
hadoop/values.yaml
image: repository:myharbor.com/bigdata/hadoop tag:3.3.2 pullPolicy:IfNotPresent ... persistence: nameNode: enabled:true storageClass:"hadoop-nn-local-storage" accessMode:ReadWriteOnce size:10Gi local: -name:hadoop-nn-0 host:"local-168-182-110" path:"/opt/bigdata/servers/hadoop/nn/data/data1" dataNode: enabled:true storageClass:"hadoop-dn-local-storage" accessMode:ReadWriteOnce size:20Gi local: -name:hadoop-dn-0 host:"local-168-182-110" path:"/opt/bigdata/servers/hadoop/dn/data/data1" -name:hadoop-dn-1 host:"local-168-182-110" path:"/opt/bigdata/servers/hadoop/dn/data/data2" -name:hadoop-dn-2 host:"local-168-182-110" path:"/opt/bigdata/servers/hadoop/dn/data/data3" -name:hadoop-dn-3 host:"local-168-182-111" path:"/opt/bigdata/servers/hadoop/dn/data/data1" -name:hadoop-dn-4 host:"local-168-182-111" path:"/opt/bigdata/servers/hadoop/dn/data/data2" -name:hadoop-dn-5 host:"local-168-182-111" path:"/opt/bigdata/servers/hadoop/dn/data/data3" -name:hadoop-dn-6 host:"local-168-182-112" path:"/opt/bigdata/servers/hadoop/dn/data/data1" -name:hadoop-dn-7 host:"local-168-182-112" path:"/opt/bigdata/servers/hadoop/dn/data/data2" -name:hadoop-dn-8 host:"local-168-182-112" path:"/opt/bigdata/servers/hadoop/dn/data/data3" ... service: nameNode: type:NodePort ports: dfs:9000 webhdfs:9870 nodePorts: dfs:30900 webhdfs:30870 dataNode: type:NodePort ports: dfs:9000 webhdfs:9864 nodePorts: dfs:30901 webhdfs:30864 resourceManager: type:NodePort ports: web:8088 nodePorts: web:30088 ... securityContext: runAsUser:9999 privileged:true
hadoop/templates/hdfs/hdfs-nn-pv.yaml
{{-range.Values.persistence.nameNode.local}}
---
apiVersion:v1
kind:PersistentVolume
metadata:
name:{{.name}}
labels:
name:{{.name}}
spec:
storageClassName:{{$.Values.persistence.nameNode.storageClass}}
capacity:
storage:{{$.Values.persistence.nameNode.size}}
accessModes:
-ReadWriteOnce
local:
path:{{.path}}
nodeAffinity:
required:
nodeSelectorTerms:
-matchExpressions:
-key:kubernetes.io/hostname
operator:In
values:
-{{.host}}
---
{{-end}}
hadoop/templates/hdfs/hdfs-dn-pv.yaml
{{-range.Values.persistence.dataNode.local}}
---
apiVersion:v1
kind:PersistentVolume
metadata:
name:{{.name}}
labels:
name:{{.name}}
spec:
storageClassName:{{$.Values.persistence.dataNode.storageClass}}
capacity:
storage:{{$.Values.persistence.dataNode.size}}
accessModes:
-ReadWriteOnce
local:
path:{{.path}}
nodeAffinity:
required:
nodeSelectorTerms:
-matchExpressions:
-key:kubernetes.io/hostname
operator:In
values:
-{{.host}}
---
{{-end}}
修改 hdfs service
mvhadoop/templates/hdfs/hdfs-nn-svc.yamlhadoop/templates/hdfs/hdfs-nn-svc-headless.yaml mvhadoop/templates/hdfs/hdfs-dn-svc.yamlhadoop/templates/hdfs/hdfs-dn-svc-headless.yaml #注意修改名稱，不要重復(fù)
hadoop/templates/hdfs/hdfs-nn-svc.yaml
#AheadlessservicetocreateDNSrecords
apiVersion:v1
kind:Service
metadata:
name:{{include"hadoop.fullname".}}-hdfs-nn
labels:
app.kubernetes.io/name:{{include"hadoop.name".}}
helm.sh/chart:{{include"hadoop.chart".}}
app.kubernetes.io/instance:{{.Release.Name}}
app.kubernetes.io/component:hdfs-nn
spec:
ports:
-name:dfs
port:{{.Values.service.nameNode.ports.dfs}}
protocol:TCP
nodePort:{{.Values.service.nameNode.nodePorts.dfs}}
-name:webhdfs
port:{{.Values.service.nameNode.ports.webhdfs}}
nodePort:{{.Values.service.nameNode.nodePorts.webhdfs}}
type:{{.Values.service.nameNode.type}}
selector:
app.kubernetes.io/name:{{include"hadoop.name".}}
app.kubernetes.io/instance:{{.Release.Name}}
app.kubernetes.io/component:hdfs-nn
hadoop/templates/hdfs/hdfs-dn-svc.yaml
#AheadlessservicetocreateDNSrecords
apiVersion:v1
kind:Service
metadata:
name:{{include"hadoop.fullname".}}-hdfs-dn
labels:
app.kubernetes.io/name:{{include"hadoop.name".}}
helm.sh/chart:{{include"hadoop.chart".}}
app.kubernetes.io/instance:{{.Release.Name}}
app.kubernetes.io/component:hdfs-nn
spec:
ports:
-name:dfs
port:{{.Values.service.dataNode.ports.dfs}}
protocol:TCP
nodePort:{{.Values.service.dataNode.nodePorts.dfs}}
-name:webhdfs
port:{{.Values.service.dataNode.ports.webhdfs}}
nodePort:{{.Values.service.dataNode.nodePorts.webhdfs}}
type:{{.Values.service.dataNode.type}}
selector:
app.kubernetes.io/name:{{include"hadoop.name".}}
app.kubernetes.io/instance:{{.Release.Name}}
app.kubernetes.io/component:hdfs-dn
修改 yarn service
mvhadoop/templates/yarn/yarn-nm-svc.yamlhadoop/templates/yarn/yarn-nm-svc-headless.yaml mvhadoop/templates/yarn/yarn-rm-svc.yamlhadoop/templates/yarn/yarn-rm-svc-headless.yaml mvhadoop/templates/yarn/yarn-ui-svc.yamlhadoop/templates/yarn/yarn-rm-svc.yaml #注意修改名稱，不要重復(fù)
hadoop/templates/yarn/yarn-rm-svc.yaml
#Servicetoaccesstheyarnwebui
apiVersion:v1
kind:Service
metadata:
name:{{include"hadoop.fullname".}}-yarn-rm
labels:
app.kubernetes.io/name:{{include"hadoop.name".}}
helm.sh/chart:{{include"hadoop.chart".}}
app.kubernetes.io/instance:{{.Release.Name}}
app.kubernetes.io/component:yarn-rm
spec:
ports:
-port:{{.Values.service.resourceManager.ports.web}}
name:web
nodePort:{{.Values.service.resourceManager.nodePorts.web}}
type:{{.Values.service.resourceManager.type}}
selector:
app.kubernetes.io/name:{{include"hadoop.name".}}
app.kubernetes.io/instance:{{.Release.Name}}
app.kubernetes.io/component:yarn-rm
修改控制器
在所有控制中新增如下內(nèi)容：
containers:
...
securityContext:
runAsUser:{{.Values.securityContext.runAsUser}}
privileged:{{.Values.securityContext.privileged}}
hadoop/templates/hadoop-configmap.yaml
###1、將/root換成/opt/apache
###2、TMP_URL="http://{{include"hadoop.fullname".}}-yarn-rm-headless:8088/ws/v1/cluster/info"
3）開始安裝
#創(chuàng)建存儲目錄
mkdir-p/opt/bigdata/servers/hadoop/{nn,dn}/data/data{1..3}
helminstallhadoop./hadoop-nhadoop--create-namespace
NOTES
NAME:hadoop LASTDEPLOYED:SatSep2417552022 NAMESPACE:hadoop STATUS:deployed REVISION:1 TESTSUITE:None NOTES: 1.YoucancheckthestatusofHDFSbyrunningthiscommand: kubectlexec-nhadoop-ithadoop-hadoop-hdfs-nn-0--/opt/hadoop/bin/hdfsdfsadmin-report 2.Youcanlisttheyarnnodesbyrunningthiscommand: kubectlexec-nhadoop-ithadoop-hadoop-yarn-rm-0--/opt/hadoop/bin/yarnnode-list 3.Createaport-forwardtotheyarnresourcemanagerUI: kubectlport-forward-nhadoophadoop-hadoop-yarn-rm-08088:8088 Thenopentheuiinyourbrowser: openhttp://localhost:8088 4.Youcanrunincludedhadooptestslikethis: kubectlexec-nhadoop-ithadoop-hadoop-yarn-nm-0--/opt/hadoop/bin/hadoopjar/opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.2-tests.jarTestDFSIO-write-nrFiles5-fileSize128MB-resFile/tmp/TestDFSIOwrite.txt 5.Youcanlistthemapreducejobslikethis: kubectlexec-nhadoop-ithadoop-hadoop-yarn-rm-0--/opt/hadoop/bin/mapredjob-list 6.Thischartcanalsobeusedwiththezeppelinchart helminstall--namespacehadoop--sethadoop.useConfigMap=true,hadoop.configMapName=hadoop-hadoopstable/zeppelin 7.Youcanscalethenumberofyarnnodeslikethis: helmupgradehadoop--setyarn.nodeManager.replicas=4stable/hadoop Makesuretoupdatethevalues.yamlifyouwanttomakethispermanent.
查看
kubectlgetpods,svc-nhadoop-owide
hdfs web：
http://192.168.182.110:30870/
yarn web：
http://192.168.182.110:30088/
5）測試驗(yàn)證
HDFS 測試驗(yàn)證
kubectlexec-ithadoop-hadoop-hdfs-nn-0-nhadoop--bash [root@local-168-182-110hadoop]#kubectlexec-ithadoop-hadoop-hdfs-nn-0-nhadoop--bash bash-4.2$ bash-4.2$ bash-4.2$hdfsdfs-mkdir/tmp bash-4.2$hdfsdfs-ls/ Found1items drwxr-xr-x-adminsupergroup02022-09-2417:56/tmp bash-4.2$echo"testhadoop">test.txt bash-4.2$hdfsdfs-puttest.txt/tmp/ bash-4.2$hdfsdfs-ls/tmp/ Found1items -rw-r--r--3adminsupergroup122022-09-2417:57/tmp/test.txt bash-4.2$hdfsdfs-cat/tmp/ cat:`/tmp':Isadirectory bash-4.2$hdfsdfs-cat/tmp/test.txt testhadoop bash-4.2$
Yarn 的測試驗(yàn)證等后面講到 hive on k8s 再來測試驗(yàn)證。
6）卸載
helmuninstallhadoop-nhadoop
kubectldeletepod-nhadoop`kubectlgetpod-nhadoop|awk'NR>1{print$1}'`--force
kubectlpatchnshadoop-p'{"metadata":{"finalizers":null}}'
kubectldeletenshadoop--force
這里也提供 git 下載地址，有需要的小伙伴可以下載部署玩玩：
https://gitee.com/hadoop-bigdata/hadoop-on-k8s
在 k8s 集群中 yarn 會慢慢被弱化，直接使用 k8s 資源調(diào)度，而不再使用 yarn 去調(diào)度資源了，這里只是部署了單點(diǎn)，僅限于測試環(huán)境使用，下一篇文章會講 Hadoop 高可用 on k8s 實(shí)現(xiàn)，請小伙伴耐心等待，有任何疑問歡迎給我留言~
審核編輯：彭靜
聲明：本文內(nèi)容及配圖由入駐作者撰寫或者入駐合作網(wǎng)站授權(quán)轉(zhuǎn)載。文章觀點(diǎn)僅代表作者本人，不代表電子發(fā)燒友網(wǎng)立場。文章及其配圖僅供工程師學(xué)習(xí)之用，如有內(nèi)容侵權(quán)或者其他違規(guī)問題，請聯(lián)系本站處理。
舉報(bào)投訴
-
硬件
+關(guān)注
關(guān)注
11文章
3328瀏覽量
66228 -
Hadoop
+關(guān)注
關(guān)注
1文章
90瀏覽量
15986 -
計(jì)算框架
+關(guān)注
關(guān)注
0文章
4瀏覽量
1932
原文標(biāo)題：7 張圖入門 Hadoop 在 K8S 環(huán)境中部署
文章出處：【微信號：magedu-Linux，微信公眾號：馬哥Linux運(yùn)維】歡迎添加關(guān)注！文章轉(zhuǎn)載請注明出處。
發(fā)布評論請先 登錄
相關(guān)推薦
hadoop無法訪問50070端口解決方案
Hadoop50070是hdfs的web管理頁面，在搭建Hadoop集群環(huán)境時(shí)，有些大數(shù)據(jù)開發(fā)技術(shù)人員會遇到Hadoop 50070端口打不開的情況，引起該問題的原因很多，想要解決這個(gè)
發(fā)表于 04-10 16:02
Hadoop的集群環(huán)境部署說明
步驟，那么多的指令肯定是會覺得很繁瑣的。畢竟都是從菜鳥一步一步過來的，記得當(dāng)初做hadoop集群環(huán)境搭建真的是很煩瑣。目前國內(nèi)的hadoop商業(yè)發(fā)行版雖然比較多，但是在集群環(huán)境的搭建方面基本都很類似
發(fā)表于 10-12 15:51
hadoop和spark的區(qū)別
學(xué)習(xí)hadoop已經(jīng)有很長一段時(shí)間了，好像是二三月份的時(shí)候朋友給了一個(gè)國產(chǎn)Hadoop發(fā)行版下載地址，因?yàn)檫€是在學(xué)習(xí)階段就下載了一個(gè)三節(jié)點(diǎn)的學(xué)習(xí)版玩一下。在研究、學(xué)習(xí)hadoop的朋友
發(fā)表于 11-30 15:51
大數(shù)據(jù)hadoop入門之hadoop家族產(chǎn)品詳解
Spark和Strom數(shù)據(jù)存在內(nèi)存中 Pig/Hive（Hadoop編程）：角色描述Pig是一種高級編程語言，在處理半結(jié)構(gòu)化數(shù)據(jù)上擁有非常高的性能，可以幫助我們縮短開發(fā)周期。Hive是數(shù)據(jù)分析查詢工具，尤其
發(fā)表于 12-26 15:02
Hadoop新手篇：hadoop入門基礎(chǔ)教程
一起就算是hadoop新手入門的一個(gè)基礎(chǔ)性教程吧（持續(xù)更新中）。 五篇文章講什么？前兩周時(shí)間寫的五篇文章，其實(shí)都在講一件事情——hadoop運(yùn)行環(huán)境安裝部署！可能口頭描述幾分鐘就可以把整個(gè)過程說完了，但
發(fā)表于 01-09 15:39
如何將程序很好的部署
我們要開啟一個(gè)項(xiàng)目，如何將程序很好的部署關(guān)系到后續(xù)的代碼移植和代碼量增多之后的管理。本文就針對上述問題，提出一種合理的工程部署方法。我們基于cobemx的軟件靈活性，確保每次編譯之后否不改變系統(tǒng)
發(fā)表于 08-24 07:40
如何將AI模型部署到嵌入式系統(tǒng)中
本期我們分享主題是如何將 AI 模型部署到嵌入式系統(tǒng)中，下一期將介紹如何在 RT-Thread 操作系統(tǒng)上運(yùn)行 Mnist Demo（手寫數(shù)字識別）。 嵌入式關(guān)聯(lián) AIAI落地一直是一
發(fā)表于 12-14 07:55
淺析Hadoop集群硬件選擇
Hadoop遠(yuǎn)遠(yuǎn)不止HDFS和MapReduce/Spark，它是一個(gè)全面的數(shù)據(jù)平臺。CDH平臺包含了很多Hadoop生態(tài)圈的其他組件。我們在做群集規(guī)劃的時(shí)候往往還需要考慮HBase，Impala和Solr等。它們都會運(yùn)行在DataNode
發(fā)表于 11-09 11:59
?1693次閱讀
基于Hadoop的I/O硬件壓縮加速器
，因此使用硬件壓縮加速器來替換軟件壓縮。Hadoop運(yùn)行在Java虛擬機(jī)上，無法直接調(diào)用底層I/O硬件壓縮加速器。通過實(shí)現(xiàn)Hadoop壓縮器／解壓縮器類和設(shè)計(jì)C++動態(tài)鏈接庫來解決從
發(fā)表于 11-27 10:49
?0次下載
idea如何將項(xiàng)目部署到tomcat服務(wù)器上
將項(xiàng)目部署到Tomcat服務(wù)器上是一個(gè)常見的操作，下面是一個(gè)詳細(xì)的步驟指南，描述了如何將項(xiàng)目成功部署到Tomcat服務(wù)器
工商網(wǎng)監(jiān)
評論