rabbitmq高可用集群搭建

需求分析基本情況

在進行RabbitMQ搭建時，我們基于現(xiàn)有的連接數(shù)據(jù)和業(yè)務(wù)需求進行了深入分析。目前的統(tǒng)計數(shù)據(jù)顯示，連接數(shù)為631，隊列數(shù)為80418。為了確保業(yè)務(wù)需求的順利滿足，我們需要在云產(chǎn)品和自建RabbitMQ消息隊列服務(wù)之間做出選擇。

經(jīng)過比較發(fā)現(xiàn)，即使選擇騰訊云的最高規(guī)格配置，其Queue數(shù)也難以滿足我們的需求，并且成本相對較高。因此，我們決定搭建自建服務(wù)。為此，計劃使用三臺配置為8核 16GB 100GB 5Mbps / 標準型SA5的服務(wù)器節(jié)點，構(gòu)建一個高可靠性集群，以確保系統(tǒng)的穩(wěn)定性和可靠性。
騰訊云:

自建服務(wù):

需求變動:
前期功能業(yè)務(wù)體諒小基于目前的現(xiàn)狀考慮，并且不影響未來的擴容的情況下的方案節(jié)點規(guī)格收容4核8G內(nèi)150GB硬(50G系統(tǒng)盤+100G數(shù)據(jù)盤)/標準型SA5,以及搭建實現(xiàn)和優(yōu)化需求：
1、 集群建設(shè)
2、 實現(xiàn)高可用
3、 節(jié)點只運行rabbitmq,所以內(nèi)存閥值調(diào)制總在比的70%

rabbimtmq集群搭建

系統(tǒng)均使用CentOS7.9

騰訊云申請三臺實例節(jié)點

初始化三臺實例主機

hostnamectl set-hostname POS_Rabbitmq_1
bash init.sh

init.sh腳步內(nèi)容,騰訊云內(nèi)置了自己的yum源,可以不需要替換

yum clean all && yum makecache
yum install  telnet curl wget lrzsz net-tools vim unzip zip  htop tree -y 

echo "=====系統(tǒng)環(huán)境初始化腳本====="
echo "1.關(guān)閉防火墻與SELinux"
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i '/SELINUX/{s/enforcing/disabled/}' /etc/selinux/config

echo "2.設(shè)置系統(tǒng)最大打開文件數(shù)"
if ! grep "* soft nofile 65535" /etc/security/limits.conf &>/dev/null; then
cat >> /etc/security/limits.conf << EOF
* soft nofile 65535   #軟限制
* hard nofile 65535   #硬限制
EOF
fi

echo "3.系統(tǒng)內(nèi)核優(yōu)化"
cat >> /etc/sysctl.conf << EOF
net.ipv4.tcp_syncookies = 1             #防范SYN洪水攻擊，0為關(guān)閉
net.ipv4.tcp_max_tw_buckets = 20480     #此項參數(shù)可以控制TIME_WAIT套接字的最大數(shù)量，避免Squid服務(wù)器被大量的TIME_WAIT套接字拖死
net.ipv4.tcp_max_syn_backlog = 20480    #表示SYN隊列的長度，默認為1024，加大隊列長度為8192，可以容納更多等待連接的網(wǎng)絡(luò)連接數(shù)
net.core.netdev_max_backlog = 262144    #每個網(wǎng)絡(luò)接口 接受數(shù)據(jù)包的速率比內(nèi)核處理這些包的速率快時，允許發(fā)送到隊列的數(shù)據(jù)包的最大數(shù)目
net.ipv4.tcp_fin_timeout = 20           #FIN-WAIT-2狀態(tài)的超時時間，避免內(nèi)核崩潰
EOF

echo "4.減少SWAP使用"
echo "0" > /proc/sys/vm/swappiness

echo "5.安裝系統(tǒng)性能分析工具及其他"
yum install -y gcc make autoconf vim sysstat net-tools iostat  lrzsz

格式化數(shù)據(jù)磁盤

數(shù)據(jù)盤默認給的是一個空盤需要直接格式化在掛載,在對安全數(shù)據(jù)要求比較嚴苛的環(huán)境中可以組RAID,這里直接格式化掛載

mkfs.ext4 /dev/vdb 
mount /dev/vdb /data
echo "/dev/vdb /data ext4 defaults 0 0" >> /etc/fstab
mount -a

mkdir -p   /data/{apd,logs,prog,setup,backup,www}
tee  /data/README.md << EOF
/data/
|-- apd         數(shù)據(jù)目錄入口
|-- backup      數(shù)據(jù)緩存目錄
|-- logs        日志目錄
|-- prog        應(yīng)用程序目錄
|-- setup       程序下載目錄
|-- www         網(wǎng)站的存放目錄
EOF

安裝docker,compose

三臺實例主機安裝docker、docker-compose 版本18.03.1、1.29.2

# step 1: 安裝必要的一些系統(tǒng)工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 git htop

# Step 2: 添加軟件源信息
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# Step 3
sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo

# Step 4: 更新并安裝Docker-CE
sudo yum makecache fast
yum -y install docker-ce-18.03.1.ce

# 安裝指定版本的Docker-CE:
# Step 1: 查找Docker-CE的版本:
# yum list docker-ce.x86_64 --showduplicates | sort -r
# Step2: 安裝指定版本的Docker-CE: (VERSION例如上面的17.03.0.ce.1-1.el7.centos)
# sudo yum -y install docker-ce-[VERSION]

# Step 5: 設(shè)置開機自啟并且啟動docker服務(wù)
systemctl enable --now docker

配置docker鏡像加速器

mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://rbmo5xql.mirror.aliyuncs.com"],
  "log-driver":"json-file",
  "bip": "192.168.1.5/24",
  "log-opts": { "max-size": "50m", "max-file": "1" }
}
EOF
 
systemctl daemon-reload && systemctl restart docker

下載docker-compose

cd /data/setup

wget -O  https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64

mv docker-compose-Linux-x86_64  docker-compose

chmod +x docker-compose 

cp /data/setup/docker-compose /usr/local/bin/

ln -sf /usr/local/bin/docker-compose /usr/bin/docker-compose

docker-compose -v
# docker-compose version 1.29.2, build 5becea4c

部署rabbitmq集群

使用rabbitmq:3.8-management鏡像,rabbitmq:3.8-management-apline包有高危漏洞[hub.docker.com官網(wǎng)查詢](https://hub.docker.com/_/rabbitmq/tags?page=&page_size=&ordering=&name=3.8-managemen)

Step1 三臺主機拉取rabbitmq鏡像

[root@pos_rabbitmq_1 /data/setup/public/rabbitmq/mq_1] eth0 = 172.17.80.27
# docker pull rabbitmq:3.8-management

[root@pos_rabbitmq_2 /data/setup/public/rabbitmq/mq_2] eth0 = 172.17.80.32
# docker pull rabbitmq:3.8-management

[root@pos_rabbitmq_3 /data/setup/public/rabbitmq/mq_3] eth0 = 172.17.80.6
# docker pull rabbitmq:3.8-management

Step2 獲取cookie

之前cookie可以在獲取后寫入docker-comose_env中定義,但是被該方法以被棄用,所以使用掛載的方式

# Step 2: 獲取cookie
[root@pos_rabbitmq_3 /data/setup/public/rabbitmq/mq_3] eth0 = 172.17.80.6
# cat > rabbitmq-cookie.sh << eof
docker run -d --name mq rabbitmq:3.8-management
sleep 10
docker exec -it mq cat /var/lib/rabbitmq/.erlang.cookie > .erlang.cookie
chmod 600 .erlang.cookie
docker rm -f mq
docker volume prune
eof

[root@pos_rabbitmq_3 /data/setup/public/rabbitmq/mq_3] eth0 = 172.17.80.6
# sh rabbitmq-cookie.sh

Step3 rabbitmq搭建集群配置文件

不適用guest用戶,使用節(jié)點模式加入集群,rabbit@pos_rabbitmq_1,在docker- compose中必須定義pos_rabbitmq_1映射IP,否則無法解析找不到節(jié)點

[root@pos_rabbitmq_3 /data/setup/public/rabbitmq/mq_3] eth0 = 172.17.80.6
# cat > rabbitmq.conf << eof
loopback_users.guest = false
listeners.tcp.default = 5672
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = rabbit@pos_rabbitmq_1
cluster_formation.classic_config.nodes.2 = rabbit@pos_rabbitmq_2
cluster_formation.classic_config.nodes.3 = rabbit@pos_rabbitmq_3
eof

Step4 docker-compose

rabbitmq內(nèi)存使用率默認占比總內(nèi)存的40%,這里需要修改為70%,env中使用RABBITMQ_VM_MEMORY_HIGH_WATERMARK定義設(shè)置

pos_rabbitmq_1 中docker-compose.yaml文件

version: "3.6"
services:
 pos_rabbitmq_1:
    image: rabbitmq:3.8-management
    restart: always
    container_name: pos_rabbitmq_1  #每個節(jié)點名稱修改即可
    network_mode: host
    extra_hosts:
      - "pos_rabbitmq_1:172.17.80.27"
      - "pos_rabbitmq_2:172.17.80.32"
      - "pos_rabbitmq_3:172.17.80.6"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /data/apd/rabbitmq:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
      - .erlang.cookie:/var/lib/rabbitmq/.erlang.cookie
      - ../enabled_plugins:/etc/rabbitmq/enabled_plugins 
      - /data/logs/rabbitmq:/var/log/rabbitmq
    environment:
      - LANG=C.UTF-8
      - RABBITMQ_DEFAULT_USER=root
      - RABBITMQ_DEFAULT_PASS=xxxxxx
      - RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.7 #內(nèi)存默認閥值設(shè)置

pos_rabbitmq_2 中docker-compose.yaml文件

version: "3.6"
services:
 pos_rabbitmq_1:
    image: rabbitmq:3.8-management
    restart: always
    container_name: pos_rabbitmq_2  #每個節(jié)點名稱修改即可
    network_mode: host
    extra_hosts:
      - "pos_rabbitmq_1:172.17.80.27"
      - "pos_rabbitmq_2:172.17.80.32"
      - "pos_rabbitmq_3:172.17.80.6"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /data/apd/rabbitmq:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
      - .erlang.cookie:/var/lib/rabbitmq/.erlang.cookie
      - ../enabled_plugins:/etc/rabbitmq/enabled_plugins 
      - /data/logs/rabbitmq:/var/log/rabbitmq
    environment:
      - LANG=C.UTF-8
      - RABBITMQ_DEFAULT_USER=root
      - RABBITMQ_DEFAULT_PASS=xxxxxx
      - RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.7

pos_rabbitmq_3 中docker-compose.yaml文件

version: "3.6"
services:
 pos_rabbitmq_1:
    image: rabbitmq:3.8-management
    restart: always
    container_name: pos_rabbitmq_3  #每個節(jié)點名稱修改即可
    network_mode: host
    extra_hosts:
      - "pos_rabbitmq_1:172.17.80.27"
      - "pos_rabbitmq_2:172.17.80.32"
      - "pos_rabbitmq_3:172.17.80.6"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /data/apd/rabbitmq:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
      - .erlang.cookie:/var/lib/rabbitmq/.erlang.cookie
      - ../enabled_plugins:/etc/rabbitmq/enabled_plugins 
      - /data/logs/rabbitmq:/var/log/rabbitmq
    environment:
      - LANG=C.UTF-8
      - RABBITMQ_DEFAULT_USER=root
      - RABBITMQ_DEFAULT_PASS=xxxxxx
      - RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.7

Step5 啟動集群,啟動完成后逐步啟動過mq2,mq3

# Step 4: 啟動集群,啟動完成后逐步啟動過mq2,mq3
[root@pos_rabbitmq_1 /data/setup/public/rabbitmq/mq_1] eth0 = 172.17.80.27
# docker-compose up -d
Creating pos_rabbitmq_1 ... done


[root@pos_rabbitmq_1 /data/setup/public/rabbitmq/mq_1] eth0 = 172.17.80.27
# docker logs pos_rabbitmq_1 -f
2024-07-30 1054.440 [info] <0.596.0> Server startup complete; 9 plugins started.
 * rabbitmq_federation_management
 * rabbitmq_federation
 * rabbitmq_web_stomp
 * rabbitmq_stomp
 * rabbitmq_web_mqtt
 * rabbitmq_mqtt
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
 completed with 9 plugins.
2024-07-30 1054.440 [info] <0.596.0> Resetting node maintenance status

Step6 集群高可用鏡像ha,任意節(jié)點執(zhí)行

[root@pos_rabbitmq_1 /data/setup/public/rabbitmq/mq_1] eth0 = 172.17.80.27
# docker exec -it pos_rabbitmq_1 /bin/bash
root@pos_rabbitmq_1:/# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all"}" with priority "0" for vhost "/" ...
root@pos_rabbitmq_1:/# exit
exit

每個節(jié)點目錄結(jié)構(gòu)及其enabled_plugins安裝插件情況如下:

目錄結(jié)構(gòu)
[root@pos_rabbitmq_1 /data/setup/public/rabbitmq] eth0 = 172.17.80.27
# tree -a
.
|-- enabled_plugins
|-- mq_1
|   |-- docker-compose.yml
|   |-- .erlang.cookie
|   |-- .rabbitmq.conf
|   `-- rabbitmq.conf
|-- mq_2
|   |-- docker-compose.yml
|   |-- .erlang.cookie
|   `-- rabbitmq.conf
|-- mq_3
|   |-- docker-compose.yml
|   |-- .erlang.cookie
|   `-- rabbitmq.conf
`-- README.md

# cat enabled_plugins
[rabbitmq_federation_management,rabbitmq_management,rabbitmq_mqtt,rabbitmq_web_mqtt,rabbitmq_stomp,rabbitmq_web_stomp].

測試

1.集群建設(shè)

pos_rabbitmq_1、pos_rabbitmq_2、pos_rabbitmq_3以組成集群

2.實現(xiàn)高可用

集群實現(xiàn)ha鏡像高可用,創(chuàng)建隊列,鏡像備份mq2,mq3

3.節(jié)點只運行rabbitmq,所以內(nèi)存閥值調(diào)制總在比的70%

總運行內(nèi)存8G,占比70% 可用5.2GB

來源：https://blog.csdn.net/qq_40189664/article/details/141823785