Centos 7.9 基于二進制文件部署kubernetes v1.25.5集群

簡述

Kubernetes（簡稱為：k8s）是Google在2014年6月開源的一個容器集群管理系統，使用Go語言開發，用于管理云平臺中多個主機上的容器化的應用，Kubernetes的目標是讓部署容器化的應用簡單并且高效,Kubernetes提供了資源調度、部署管理、服務發現、擴容縮容、監控，維護等一整套功能，努力成為跨主機集群的自動部署、擴展以及運行應用程序容器的平臺。它支持一系列容器工具, 包括Docker、Containerd等。

一、集群環境準備

1.1 主機規劃

服務器系統	節點IP	主機角色	CUP/內存	Hostname	內核	軟件列表
Centos 7.9.2009	192.168.1.92	master	2C/4G	k8s-master1	6.1.0	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc
Centos 7.9.2009	192.168.1.93	master	2C/4G	k8s-master2	6.1.0	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc
Centos 7.9.2009	192.168.1.94	master	2C/4G	k8s-master3	6.1.0	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc
Centos 7.9.2009	192.168.1.96	worker	2C/4G	k8s-worker1	6.1.0	kubelet、kube-proxy、Containerd、runc
Centos 7.9.2009	192.168.1.111	LB	1C/1G	ha1	6.1.0	haproxy、keepalived
Centos 7.9.2009	192.168.1.112	LB	1C/1G	ha2	6.1.0	haproxy、keepalived
Centos 7.9.2009	192.168.1.100	VIP(虛擬IP)	/	/	/	/

1.2 軟件版本

軟件名稱	版本	備注
CentOS 7.9	kernel版本：6.1.0	升級后版本
kubernetes	v1.25.5
etcd	v3.5.6	最新版本
calico	v3.23	穩定版本
coredns	1.8.4
containerd	1.6.12	最新版本
runc	1.1.4	最新版本
haproxy	1.5.18	yum源默認
keepalived	v1.3.5	yum源默認

1.3 網絡分配

網絡名稱	網段	備注
Node網絡	192.168.1.0/24
Service網絡	10.96.0.0/16
Pod網絡	10.244.0.0/16

二、集群部署

2.1主機準備

2.1.1 主機名設置

注：關于主機名參見1.1小節主機規劃表

hostnamectl set-hostname ha1
hostnamectl set-hostname ha2
hostnamectl set-hostname k8s-master1
hostnamectl set-hostname k8s-master2
hostnamectl set-hostname k8s-master3
hostnamectl set-hostname k8s-worker1

2.1.2 主機與IP地址解析

cat >> /etc/hosts << EOF
192.168.1.111 ha1
192.168.1.112 ha2
192.168.1.92 k8s-master1
192.168.1.93 k8s-master2
192.168.1.94 k8s-master3
192.168.1.96 k8s-worker1
EOF

2.1.3 主機安全設置

2.1.3.1 關閉防火墻

systemctl stop firewalld
systmctl disable firewalld
firewall-cmd --state

2.1.3.2 關閉selinux

setenforce 0
sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
sestatus

2.1.4 交換分區設置

swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
echo "vm.swappiness=0" >> /etc/sysctl.conf
sysctl -p

2.1.5 主機系統時間同步

2.1.5.1 安裝軟件

yum -y install ntpdate

2.1.5.2 制定時間同步計劃任務(每小時執行一次同步)

crontab -e
0 */1 * * * ntpdate time1.aliyun.com

2.1.6 主機系統優化

limit優化

ulimit -SHn 65535

cat <<EOF >> /etc/security/limits.conf
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

2.1.7 ipvs管理工具安裝及模塊加載

為集群節點安裝，負載均衡節點不用安裝

yum -y install ipvsadm ipset sysstat conntrack libseccomp

臨時加入，所有節點配置ipvs模塊，在內核4.19+版本nf_conntrack_ipv4已經改為nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可：

modprobe -- ip_vs 
modprobe -- ip_vs_rr 
modprobe -- ip_vs_wrr 
modprobe -- ip_vs_sh 
modprobe -- nf_conntrack

永久加入，創建 /etc/modules-load.d/ipvs.conf 并加入以下內容：

cat >/etc/modules-load.d/ipvs.conf <<EOF 
ip_vs 
ip_vs_lc 
ip_vs_wlc 
ip_vs_rr 
ip_vs_wrr 
ip_vs_lblc 
ip_vs_lblcr 
ip_vs_dh 
ip_vs_sh 
ip_vs_fo 
ip_vs_nq 
ip_vs_sed 
ip_vs_ftp 
ip_vs_sh 
nf_conntrack 
ip_tables 
ip_set 
xt_set 
ipt_set 
ipt_rpfilter 
ipt_REJECT 
ipip 
EOF

2.1.8 加載containerd相關內核模塊

注：master和worker節點執行

2.1.8.1 臨時加載模塊

modprobe overlay
modprobe br_netfilter

2.1.8.2 永久性加載模塊

cat > /etc/modules-load.d/containerd.conf << EOF
overlay
br_netfilter
EOF

2.1.8.3 設置為開機啟動

systemctl enable --now systemd-modules-load.service

2.1.9 Linux內核升級

在所有節點中安裝,需要重新操作系統更換內核。

[root@localhost ~]# yum -y install perl
[root@localhost ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[root@localhost ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
[root@localhost ~]# yum  --enablerepo="elrepo-kernel"  -y install kernel-ml.x86_64
[root@localhost ~]# grub2-set-default 0
[root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

2.1.10 Linux內核優化

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 131072
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

使其生效：

sysctl --system

所有節點配置完內核后，重啟服務器，保證重啟后內核依舊加載

reboot -h now

重啟后查看ipvs模塊加載情況：

lsmod | grep --color=auto -e ip_vs -e nf_conntrack

重啟后查看containerd相關模塊加載情況：

lsmod | egrep 'br_netfilter'
lsmod | egrep 'overlay'

2.1.11 其它工具安裝(選裝)

注：jq在本次集群部署中，無關緊要，如果無法安裝也不影響。

master和worker節點執行

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y

2.2 負載均衡器準備

2.2.1 安裝haproxy與keepalived

yum -y install haproxy keepalived

2.2.2 HAProxy配置

cat >/etc/haproxy/haproxy.cfg<<"EOF"
global
 maxconn 2000
 ulimit-n 16384
 log 127.0.0.1 local0 err
 stats timeout 30s

defaults
 log global
 mode http
 option httplog
 timeout connect 5000
 timeout client 50000
 timeout server 50000
 timeout http-request 15s
 timeout http-keep-alive 15s

frontend monitor-in
 bind *:33305
 mode http
 option httplog
 monitor-uri /monitor

frontend k8s-master
 bind 0.0.0.0:6443
 bind 127.0.0.1:6443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 default_backend k8s-master

backend k8s-master
 mode tcp
 option tcplog
 option tcp-check
 balance roundrobin
 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 server  k8s-master1  192.168.1.92:6443 check
 server  k8s-master2  192.168.1.93:6443 check
 server  k8s-master3  192.168.1.94:6443 check
EOF

2.2.3 KeepAlived

注：主從配置不一致，需要注意。

ha1:

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state MASTER
   interface ens32
   mcast_src_ip 192.168.1.111
   virtual_router_id 51
   priority 100
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       192.168.1.100
   }
   track_script {
      chk_apiserver
   }
}
EOF

說明：

state MASTER　　　　　　　　　　 # 主寫master,從寫backup

interface ens32　　　　　　　　　　 # 寫自己主機網卡名稱，查看：ip a

mcast_src_ip 192.168.1.111　　　　 # 寫keepalived的主機IP

priority 100　　　　　　　　　　　　 # 權重，從小于100

virtual_ipaddress 192.168.1.100　　 # ip隨意編寫，兩臺負載均衡器保持一致即可

ha2:

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
  interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state BACKUP
   interface ens32
   mcast_src_ip 192.168.1.112
   virtual_router_id 51
   priority 99
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       192.168.1.100
   }
   track_script {
      chk_apiserver
   }
}
EOF

2.2.4 健康檢查腳本

注：ha1及ha2均要配置

cat > /etc/keepalived/check_apiserver.sh <<"EOF"
#!/bin/bash
err=0
for k in $(seq 1 3)
do
   check_code=$(pgrep haproxy)
   if [[ $check_code == "" ]]; then
       err=$(expr $err + 1)
       sleep 1
       continue
   else
       err=0
       break
   fi
done

if [[ $err != "0" ]]; then
   echo "systemctl stop keepalived"
   /usr/bin/systemctl stop keepalived
   exit 1
else
   exit 0
fi
EOF

賦予腳本執行權限：

chmod +x /etc/keepalived/check_apiserver.sh

2.2.5 啟動服務并驗證

systemctl daemon-reload
systemctl enable --now haproxy keepalived
systemctl status haproxy keepalived

2.3 配置免密登錄

注：在k8s-master1上操作

2.3.1 生成密鑰：

ssh-keygen

說明：

(/root/.ssh/id_rsa):　　　　　　　　# 證書私鑰保存的位置

Enter passphrase：　　　　　　　　# 私鑰使用過程中設置的密碼

Enter same passphrase again:　　　　# 密碼的確認

三個地方全部回車即可!

2.3.2 拷貝至其他服務器

ssh-copy-id root@k8s-master1
ssh-copy-id root@k8s-master2
ssh-copy-id root@k8s-master3
ssh-copy-id root@k8s-worker1

輸入yes，然后輸入目標服務器密碼！

2.3.3 測試免密登錄

ssh root@k8s-master2

2.4 創建CA及etcd證書

注：在k8s-master1上操作

2.4.1 創建工作目錄

mkdir -p /data/k8s-work

2.4.2 獲取cfssl工具

cd /data/k8s-work
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

說明：

cfssl是使用go編寫，由CloudFlare開源的一款PKI/TLS工具。主要程序有：

- cfssl，　　是CFSSL的命令行工具

- cfssljson　　用來從cfssl程序獲取JSON輸出，并將證書，密鑰，CSR和bundle寫入文件中。

賦予權限：

chmod +x cfssl*

剪切工具至系統目錄：

mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

驗證：

cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6

2.4.3 創建CA證書

2.4.3.1 配置ca證書請求文件

cat > ca-csr.json <<"EOF"
{
  "CN": "kubernetes",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ],
  "ca": {
          "expiry": "87600h"
  }
}
EOF

2.4.3.2 創建ca證書

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

2.4.3.3 生成配置ca證書策略

cfssl print-defaults config > ca-config.json

說明：自己生成的證書配置，"expiry":有效期是8760h(一年)，也可以修改為87600h(十年)。

或者使用以下配置：

cat > ca-config.json <<"EOF"
{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "kubernetes": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}
EOF

說明：

server auth 表示client可以對使用該ca對server提供的證書進行驗證

client auth 表示server可以使用該ca對client提供的證書進行驗證

2.4.4 創建etcd證書

2.4.4.1 配置etcd請求文件

注：

etcd可以部署到集群的master節點或worker節點，不受影響。
修改主機IP為自己的主機。

cat > etcd-csr.json <<"EOF"
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.1.92",
    "192.168.1.93",
    "192.168.1.94"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "kubemsb",
    "OU": "CN"
  }]
}
EOF

2.4.4.2 生成etcd證書

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson  -bare etcd

2.4.4.3 查看生成的etcd證書文件

[root@k8s-master1 k8s-work]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem

2.4.5 部署etcd集群

2.4.5.1 下載etcd軟件包

下載地址：https://github.com/

搜索框輸入etcd，選擇etcd-io/etcd

選擇右側releases

選擇自己需要下載的安裝包

下載安裝包：

wget https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz

2.4.5.2 安裝etcd軟件

tar xf etcd-v3.5.6-linux-amd64.tar.gz
cp -p etcd-v3.5.6-linux-amd64/etcd* /usr/local/bin/

驗證安裝：

[root@k8s-master1 k8s-work]# etcdctl version
etcdctl version: 3.5.6
API version: 3.5

2.4.5.3 分發etcd軟件

scp etcd-v3.5.6-linux-amd64/etcd* k8s-master2:/usr/local/bin/
scp etcd-v3.5.6-linux-amd64/etcd* k8s-master3:/usr/local/bin/

2.4.5.4 創建配置文件

創建目錄：

mkdir /etc/etcd

創建配置文件：

cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.92:2380,etcd2=https://192.168.1.93:2380,etcd3=https://192.168.1.94:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

說明：

ETCD_NAME：節點名稱，集群中唯一

ETCD_DATA_DIR：數據目錄

ETCD_LISTEN_PEER_URLS：集群通信監聽地址

ETCD_LISTEN_CLIENT_URLS：客戶端訪問監聽地址

ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址

ETCD_ADVERTISE_CLIENT_URLS：客戶端通告地址

ETCD_INITIAL_CLUSTER：集群節點地址

ETCD_INITIAL_CLUSTER_TOKEN：集群Token

ETCD_INITIAL_CLUSTER_STATE：加入集群的當前狀態，new是新集群，existing表示加入已有集群

2.4.5.5 創建服務配置文件

創建目錄：

mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd

拷貝證書至指定目錄：

cd /data/k8s-work
cp ca*.pem /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl

創建服務文件：

cat > /etc/systemd/system/etcd.service <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

2.4.5.6 同步etcd配置到集群其它master節點

創建目錄：

mkdir -p /etc/etcd
mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd

服務配置文件,需要修改etcd節點名稱及IP地址

for i in k8s-master2 k8s-master3 \
do \
scp /etc/etcd/etcd.conf $i:/etc/etcd/ \
done

k8s-master2:

cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.93:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.92:2380,etcd2=https://192.168.1.93:2380,etcd3=https://192.168.1.94:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

k8s-master3:

cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.94:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.94:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.94:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.94:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.92:2380,etcd2=https://192.168.1.93:2380,etcd3=https://192.168.1.94:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

同步證書文件：

for i in k8s-master2 k8s-master3 ;do scp /etc/etcd/ssl/* $i:/etc/etcd/ssl ;done

同步服務啟動配置文件：

for i in k8s-master2 k8s-master3 ;do scp /etc/systemd/system/etcd.service $i:/etc/systemd/system/ ;done

2.4.5.7 啟動etcd集群

注：如果報錯無法啟動，則需要將其他etcd節點設置完成后才可以啟動（同時操作以下啟動命令）

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

2.4.5.8 驗證集群狀態

檢查ETCD集群健康狀態

ETCDCTL_API=3 /usr/local/bin/etcdctl \
--write-out=table \
--cacert=/etc/etcd/ssl/ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.94:2379 endpoint health

結果：

+---------------------------+--------+-------------+-------+
|         ENDPOINT          | HEALTH |    TOOK     | ERROR |
+---------------------------+--------+-------------+-------+
| https://192.168.1.92:2379 |   true | 11.109782ms |       |
| https://192.168.1.93:2379 |   true | 14.648997ms |       |
| https://192.168.1.94:2379 |   true | 16.783152ms |       |
+---------------------------+--------+-------------+-------+

檢查ETCD數據庫性能

ETCDCTL_API=3 /usr/local/bin/etcdctl \
--write-out=table \
--cacert=/etc/etcd/ssl/ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.94:2379 check perf

結果：

 59 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom  !  98.33%PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.055096s
PASS: Stddev is 0.001185s
PASS

檢查ETCD集群成員列表

ETCDCTL_API=3 /usr/local/bin/etcdctl \
--write-out=table \
--cacert=/etc/etcd/ssl/ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.94:2379 member list

結果：

+------------------+---------+-------+---------------------------+---------------------------+------------+
|        ID        | STATUS  | NAME  |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------+---------------------------+---------------------------+------------+
| af05139f75a68867 | started | etcd1 | https://192.168.1.92:2380 | https://192.168.1.92:2379 |      false |
| c3bab7c20fba3f60 | started | etcd2 | https://192.168.1.93:2380 | https://192.168.1.93:2379 |      false |
| fbba12d5c6ebb577 | started | etcd3 | https://192.168.1.94:2380 | https://192.168.1.94:2379 |      false |
+------------------+---------+-------+---------------------------+---------------------------+------------+

檢查ETCD節點狀態（可以看出誰是集群的leader）

ETCDCTL_API=3 /usr/local/bin/etcdctl \
--write-out=table \
--cacert=/etc/etcd/ssl/ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.94:2379 endpoint status

結果：

+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.1.92:2379 | af05139f75a68867 |   3.5.6 |   22 MB |     false |      false |         3 |       8982 |               8982 |        |
| https://192.168.1.93:2379 | c3bab7c20fba3f60 |   3.5.6 |   22 MB |      true |      false |         3 |       8982 |               8982 |        |
| https://192.168.1.94:2379 | fbba12d5c6ebb577 |   3.5.6 |   22 MB |     false |      false |         3 |       8982 |               8982 |        |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

2.5 Kubernetes集群部署

2.5.1 Kubernetes軟件包下載

下載地址：https://github.com/kubernetes/kubernetes/releases

下載安裝包（我這里用的v1.25.5版本）：

wget https://dl.k8s.io/v1.25.5/kubernetes-server-linux-amd64.tar.gz

2.5.2 Kubernetes軟件包安裝

tar -xvf kubernetes-server-linux-amd64.tar.gz

cd kubernetes/server/bin/

cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/

2.5.3 Kubernetes軟件分發

scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master2:/usr/local/bin/
scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master3:/usr/local/bin/

根據自己需求，拷貝以下服務：

scp kubelet kube-proxy k8s-master1:/usr/local/bin
scp kubelet kube-proxy k8s-master2:/usr/local/bin
scp kubelet kube-proxy k8s-master3:/usr/local/bin
scp kubelet kube-proxy k8s-worker1:/usr/local/bin

2.5.4 在集群節點上創建目錄

注：所有節點

mkdir -p /etc/kubernetes/        
mkdir -p /etc/kubernetes/ssl     
mkdir -p /var/log/kubernetes

2.5.5 部署api-server

2.5.5.1 創建apiserver證書請求文件

注：master1操作

cat > kube-apiserver-csr.json << "EOF"
{
"CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.1.92",
    "192.168.1.93",
    "192.168.1.94",
    "192.168.1.95",
    "192.168.1.96",
    "192.168.1.97",
    "192.168.1.98",
    "192.168.1.99",
    "192.168.1.100",
    "10.96.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ]
}
EOF

說明：

如果 hosts 字段不為空則需要指定授權使用該證書的 IP（含VIP）或域名列表。由于該證書被集群使用，需要將節點的IP都填上，為了方便后期擴容可以多寫幾個預留的IP。

同時還需要填寫 service 網絡的首個IP(一般是 kube-apiserver 指定的 service-cluster-ip-range 網段的第一個IP，如 10.96.0.1)。

2.5.5.2 生成apiserver證書及token文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver

創建TLS機制所需TOKEN（自動簽發給客戶端證書的機制）:

cat > token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

說明：

創建TLS機制所需TOKEN

TLS Bootstraping：Master apiserver啟用TLS認證后，Node節點kubelet和kube-proxy與kube-apiserver進行通信，必須使用CA簽發的有效證書才可以，當Node節點很多時，這種客戶端證書頒發需要大量工作，同樣也會增加集群擴展復雜度。為了簡化流程，Kubernetes引入了TLS bootstraping機制來自動頒發客戶端證書，kubelet會以一個低權限用戶自動向apiserver申請證書，kubelet的證書由apiserver動態簽署。所以強烈建議在Node上使用這種方式，目前主要用于kubelet，kube-proxy還是由我們統一頒發一個證書。

2.5.5.3 創建apiserver服務配置文件

cat > /etc/kubernetes/kube-apiserver.conf << "EOF"
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.1.92 \
  --secure-port=6443 \
  --advertise-address=192.168.1.92 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=api \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.94:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"
EOF

說明：

--bind-address :監聽地址
--secure-port：https安全端口
--advertise-address：集群通告地址
--authorization-mode：認證授權，啟用RBAC授權和節點自管理
--allow-privileged：啟用授權
--service-cluster-ip-range：Service虛擬IP地址段
--enable-bootstrap-token-auth：啟用TLS bootstrap機制
--token-auth-file：bootstrap token文件
--service-node-port-range：Service nodeport類型默認分配端口范圍
--kubelet-https：apiserver主動訪問kubelet時默認使用https
--kubelet-client-xxx：apiserver訪問kubelet客戶端證書
--tls-xxx-file：apiserver https證書
--etcd-xxxfile：連接Etcd集群證書
--etcd-servers: etcd集群地址
--audit-log-xxx：審計日志
--logtostderr：啟用日志
--log-dir:日志目錄
--v：日志等級，越小越多

2.5.5.4 創建apiserver服務管理配置文件

注：

After 　　代表誰在前啟動
Requires 代表需要哪個服務

cat > /etc/systemd/system/kube-apiserver.service << "EOF"
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

說明：

此兩橫字段說明，要啟動kube-apiserver，必須先啟動etcd。

After=etcd.service

Wants=etcd.service

2.5.5.5 同步文件到集群master節點

拷貝目錄：

cd /data/k8s-work

拷貝ca證書到master1指定目錄:

cp ca*.pem /etc/kubernetes/ssl/

拷貝kube-apiserver證書到master1指定目錄:　

cp kube-apiserver*.pem /etc/kubernetes/ssl/

拷貝token.csv文件到master1指定目錄:

cp token.csv /etc/kubernetes/

同步token.csv文件到其他master：

scp /etc/kubernetes/token.csv k8s-master2:/etc/kubernetes
scp /etc/kubernetes/token.csv k8s-master3:/etc/kubernetes

同步kube-apiserver證書到其他master：

scp /etc/kubernetes/ssl/kube-apiserver*.pem k8s-master2:/etc/kubernetes/ssl
scp /etc/kubernetes/ssl/kube-apiserver*.pem k8s-master3:/etc/kubernetes/ssl

同步ca證書到其他master：

scp /etc/kubernetes/ssl/ca*.pem k8s-master2:/etc/kubernetes/ssl
scp /etc/kubernetes/ssl/ca*.pem k8s-master3:/etc/kubernetes/ssl

同步kube-apiserver配置文件到master2（需要修改主機IP）：

scp /etc/kubernetes/kube-apiserver.conf k8s-master2:/etc/kubernetes/kube-apiserver.conf

或者直接執行以下內容：

cat > /etc/kubernetes/kube-apiserver.conf << "EOF"
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.1.93 \
  --secure-port=6443 \
  --advertise-address=192.168.1.93 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=api \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.94:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"
EOF

同步kube-apiserver配置文件到master3（需要修改主機IP）：

scp /etc/kubernetes/kube-apiserver.conf k8s-master3:/etc/kubernetes/kube-apiserver.conf

或者直接執行以下內容：

cat > /etc/kubernetes/kube-apiserver.conf << "EOF"
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.1.94 \
  --secure-port=6443 \
  --advertise-address=192.168.1.94 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=api \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.94:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"
EOF

同步服務啟動文件（master節點）：　　

scp /etc/systemd/system/kube-apiserver.service k8s-master2:/etc/systemd/system/kube-apiserver.service

scp /etc/systemd/system/kube-apiserver.service k8s-master3:/etc/systemd/system/kube-apiserver.service

2.5.5.6 啟動apiserver服務

systemctl daemon-reload
systemctl enable --now kube-apiserver
systemctl status kube-apiserver

2.5.5.7 測試

curl --insecure https://192.168.1.92:6443/
curl --insecure https://192.168.1.93:6443/
curl --insecure https://192.168.1.94:6443/
curl --insecure https://192.168.1.100:6443/

說明：

未經授權，所有都會報401。

2.5.6 部署kubectl

2.5.6.1 創建kubectl證書請求文件

cat > admin-csr.json << "EOF"
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",             
      "OU": "system"
    }
  ]
}
EOF

說明：

后續 kube-apiserver 使用 RBAC 對客戶端(如 kubelet、kube-proxy、Pod)請求進行授權；

kube-apiserver 預定義了一些 RBAC 使用的 RoleBindings，如 cluster-admin 將 Group system:masters 與 Role cluster-admin 綁定，該 Role 授予了調用kube-apiserver 的所有 API的權限；

O指定該證書的 Group 為 system:masters，kubelet 使用該證書訪問 kube-apiserver 時，由于證書被 CA 簽名，所以認證通過，同時由于證書用戶組為經過預授權的 system:masters，所以被授予訪問所有 API 的權限；

注：

這個admin 證書，是將來生成管理員用的kubeconfig 配置文件用的，現在我們一般建議使用RBAC 來對kubernetes 進行角色權限控制， kubernetes 將證書中的CN 字段作為User， O 字段作為 Group；

"O": "system:masters", 必須是system:masters，否則后面kubectl create clusterrolebinding報錯。

2.5.6.2 生成證書文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

2.5.6.3 復制文件到指定目錄

cd /data/k8s-work
cp admin*.pem /etc/kubernetes/ssl/

2.5.6.4 生成kubeconfig配置文件

注：kube.config 為 kubectl 的配置文件，包含訪問 apiserver 的所有信息，如 apiserver 地址、CA 證書和自身使用的證書

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.100:6443 --kubeconfig=kube.config

kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config

kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config

kubectl config use-context kubernetes --kubeconfig=kube.config

說明：

第一步：kubectl 管理的集群，所對應的證書，以及證書的訪問鏈接。

第二步：配置證書的角色，管理員，使用的那一個人的證書，來對集群管理。

第三步：設置安全上下文。

第四步：使用安全上下文，對集群進行管理。

2.5.6.5 準備kubectl配置文件并進行角色綁定

mkdir ~/.kube
cp kube.config ~/.kube/config
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes --kubeconfig=/root/.kube/config

2.5.6.6 查看集群狀態

export KUBECONFIG=$HOME/.kube/config

查看集群信息

kubectl cluster-info

查看集群組件狀態（后續部署組件中都可以查看確認）

kubectl get componentstatuses

查看命名空間中資源對象

kubectl get all --all-namespaces

2.5.6.7 同步kubectl配置文件到集群其它master節點

k8s-master2:

mkdir /root/.kube

k8s-master3:

mkdir /root/.kube

master1操作同步：

scp /root/.kube/config k8s-master2:/root/.kube/config
scp /root/.kube/config k8s-master3:/root/.kube/config

2.5.6.8 配置kubectl命令補全(可選)

注：如果配置了此命令，長時間依賴補全命令，會忘記命令，不建議安裝。

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
kubectl completion bash > ~/.kube/completion.bash.inc
source '/root/.kube/completion.bash.inc'  
source $HOME/.bash_profile

2.5.7 部署kube-controller-manager

2.5.7.1 創建kube-controller-manager證書請求文件

cat > kube-controller-manager-csr.json << "EOF"
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.1.92",
      "192.168.1.93",
      "192.168.1.94"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "Beijing",
        "L": "Beijing",
        "O": "system:kube-controller-manager",
        "OU": "system"
      }
    ]
}
EOF

說明：

hosts 列表包含所有 kube-controller-manager 節點 IP；

CN 為 system:kube-controller-manager;

O 為 system:kube-controller-manager，kubernetes 內置的 ClusterRoleBindings system:kube-controller-manager 賦予 kube-controller-manager 工作所需的權限

2.5.7.2 創建kube-controller-manager證書文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

查看生成的證書：

cd /data/k8s-work/
# ls
kube-controller-manager.csr     
kube-controller-manager-csr.json
kube-controller-manager-key.pem
kube-controller-manager.pem

2.5.7.3 創建kube-controller-manager的kube-controller-manager.kubeconfig

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.100:6443 --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

2.5.7.4 創建kube-controller-manager配置文件

注：--bind-address 可以綁定主機IP，也可以寫127.0.0.1，否則每臺主機都需要修改。

cat > kube-controller-manager.conf << "EOF"
KUBE_CONTROLLER_MANAGER_OPTS=" \
  --secure-port=10257 \
  --bind-address=127.0.0.1 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.96.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.244.0.0/16 \
  --cluster-signing-duration=87600h \ 
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-sync-period=10s \
  --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2"
EOF

2.5.7.5 創建服務啟動文件

cat > kube-controller-manager.service << "EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

2.5.7.6 同步文件到集群master節點

拷貝文件到本機（master1）指定目錄：

cp kube-controller-manager*.pem /etc/kubernetes/ssl/
cp kube-controller-manager.kubeconfig /etc/kubernetes/
cp kube-controller-manager.conf /etc/kubernetes/
cp kube-controller-manager.service /usr/lib/systemd/system/

同步文件到集群master節點：

scp  kube-controller-manager*.pem k8s-master2:/etc/kubernetes/ssl/
scp  kube-controller-manager*.pem k8s-master3:/etc/kubernetes/ssl/
scp  kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master2:/etc/kubernetes/
scp  kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master3:/etc/kubernetes/
scp  kube-controller-manager.service k8s-master2:/usr/lib/systemd/system/
scp  kube-controller-manager.service k8s-master3:/usr/lib/systemd/system/

查看證書

openssl x509 -in /etc/kubernetes/ssl/kube-controller-manager.pem -noout -text

2.5.7.7 啟動服務

systemctl daemon-reload 
systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager

2.5.7.8 查看集群組件狀態（查看controller-manager）

[root@k8s-master1 k8s-work]# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get "https://127.0.0.1:10259/healthz": dial tcp 127.0.0.1:10259: connect: connection refused   
controller-manager   Healthy     ok                                                                                             
etcd-1               Healthy     {"health":"true","reason":""}                                                                  
etcd-2               Healthy     {"health":"true","reason":""}                                                                  
etcd-0               Healthy     {"health":"true","reason":""}

2.5.8 部署kube-scheduler

2.5.8.1 創建kube-scheduler證書請求文件

cat > kube-scheduler-csr.json << "EOF"
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.1.92",
      "192.168.1.93",
      "192.168.1.94"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "Beijing",
        "L": "Beijing",
        "O": "system:kube-scheduler",
        "OU": "system"
      }
    ]
}
EOF

2.5.8.2 生成kube-scheduler證書

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

查看生成的證書：

cd /data/k8s-work/
# ls
kube-scheduler.csr
kube-scheduler-csr.json
kube-scheduler-key.pem
kube-scheduler.pem

2.5.8.3 創建kube-scheduler的kubeconfig

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.100:6443 --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

2.5.8.4 創建kube-scheduler服務配置文件

cat > kube-scheduler.conf << "EOF"
KUBE_SCHEDULER_OPTS=" \
--bind-address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--leader-elect=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2"
EOF

2.5.8.5創建服務啟動配置文件

cat > kube-scheduler.service << "EOF"
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

2.5.8.6 同步文件至集群master節點

拷貝文件到本機（master1）指定目錄：

cp kube-scheduler*.pem /etc/kubernetes/ssl/
cp kube-scheduler.kubeconfig /etc/kubernetes/
cp kube-scheduler.conf /etc/kubernetes/
cp kube-scheduler.service /usr/lib/systemd/system/

同步文件到集群master節點：

scp  kube-scheduler*.pem k8s-master2:/etc/kubernetes/ssl/
scp  kube-scheduler*.pem k8s-master3:/etc/kubernetes/ssl/
scp  kube-scheduler.kubeconfig kube-scheduler.conf k8s-master2:/etc/kubernetes/
scp  kube-scheduler.kubeconfig kube-scheduler.conf k8s-master3:/etc/kubernetes/
scp  kube-scheduler.service k8s-master2:/usr/lib/systemd/system/
scp  kube-scheduler.service k8s-master3:/usr/lib/systemd/system/

2.5.8.7 啟動服務

systemctl daemon-reload
systemctl enable --now kube-scheduler
systemctl status kube-scheduler

2.5.8.8 查看集群組件狀態（查看scheduler）

[root@k8s-master1 k8s-work]# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}   
etcd-1               Healthy   {"health":"true","reason":""}   
scheduler            Healthy   ok                              
etcd-2               Healthy   {"health":"true","reason":""}

2.5.9 工作節點（worker node）部署

2.5.9.1 Containerd安裝及配置

2.5.9.1.1 獲取軟件包

下載地址：https://github.com/

選擇cri-containerd-cni-1.6.12-linux-amd64.tar.gz 類型，此類包括了容器管理，網絡插件適用

2.5.9.1.2 安裝containerd

注：次此master和worker全部安裝（master也做為運行pod的節點），根據自己需求安裝

（1）下載安裝包：

wget https://github.com/containerd/containerd/releases/download/v1.6.12/cri-containerd-cni-1.6.12-linux-amd64.tar.gz

安裝包可能無法下載，我已放到百度網盤：

鏈接：https://pan.baidu.com/s/1Ma28Aanwx5nuhGvqptUsWQ?pwd=ixok
提取碼：ixok

（2）手動上傳至master1,然后同步runc到其他三個節點（master2、master3、worker1）：

scp cri-containerd-cni-1.6.12-linux-amd64.tar.gz k8s-master2:/root/
scp cri-containerd-cni-1.6.12-linux-amd64.tar.gz k8s-master3:/root/
scp cri-containerd-cni-1.6.12-linux-amd64.tar.gz k8s-worker1:/root/

（3）四臺主機都解壓：

tar -xf cri-containerd-cni-1.6.12-linux-amd64.tar.gz -C /

說明：

默認解壓后會有如下目錄：

etc

opt

usr

會把對應的目解壓到/下對應目錄中，這樣就省去復制文件步驟。

2.5.9.1.3 生成配置文件并修改

創建目錄：

mkdir /etc/containerd

生成containerd配置文件：

containerd config default >/etc/containerd/config.toml

查看：

# ls /etc/containerd/
config.toml

下面的配置文件中已修改，可不執行，僅修改默認時執行。

sed -i 's@systemd_cgroup = false@systemd_cgroup = true@' /etc/containerd/config.toml

下面的配置文件中已修改，可不執行，僅修改默認時執行。

sed -i 's@k8s.gcr.io/pause:3.6@registry.aliyuncs.com/google_containers/pause:3.6@' /etc/containerd/config.toml

注：生成的配置文件后期改動的地方較多，缺少鏡像倉庫，這里直接換成可單機使用、也可k8s環境使用的配置文件并配置好鏡像加速器。（根據自己需求決定是否使用）

cat >/etc/containerd/config.toml<<EOF
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = -999

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
    stats_collect_period = 10
    systemd_cgroup = true
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = "/etc/cni/net.d/10-default.conf"
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = [
            "https://docker.mirrors.ustc.edu.cn",
            "http://hub-mirror.c.163.com"
          ]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = [
            "https://gcr.mirrors.ustc.edu.cn"
          ]
        [plugins.cri.registry.mirrors."k8s.gcr.io"]
          endpoint = [
            "https://gcr.mirrors.ustc.edu.cn/google-containers/"
          ]
        [plugins.cri.registry.mirrors."quay.io"]
          endpoint = [
            "https://quay.mirrors.ustc.edu.cn"
          ]
        [plugins.cri.registry.mirrors."harbor.kubemsb.com"]
          endpoint = [
            "http://harbor.kubemsb.com"
          ]
    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"
EOF

2.5.9.1.4 安裝runc

注：

由于上述軟件包中包含的runc對系統依賴過多，所以建議單獨下載安裝。

默認runc執行時提示：runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond

獲取runc

下載地址：https://github.com/

(1) 下載安裝包：

wget https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64

安裝包可能無法下載，我已放到百度網盤：

鏈接：https://pan.baidu.com/s/1Ma28Aanwx5nuhGvqptUsWQ?pwd=ixok
提取碼：ixok

(2) 賦予runc執行權限：

chmod +x runc.amd64

(3) 替換掉原軟件包中的runc：

mv runc.amd64 /usr/local/sbin/runc

(4) 同步runc到其他三個節點（master2、master3、worker1）

scp /usr/local/sbin/runc k8s-master2:/usr/local/sbin/runc
scp /usr/local/sbin/runc k8s-master3:/usr/local/sbin/runc
scp /usr/local/sbin/runc k8s-worker1:/usr/local/sbin/runc

(5) 驗證：

[root@k8s-master1 k8s-work]# runc -v
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d1
spec: 1.0.2-dev
go: go1.17.10
libseccomp: 2.5.4

(6) 啟動Containerd

systemctl enable containerd
systemctl start containerd
systemctl status containerd

2.5.9.2 部署kubelet

注：在k8s-master1上操作

2.5.9.2.1 創建kubelet-bootstrap.kubeconfig

BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.100:6443 --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

確定集群的角色和資源調用

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=kubelet-bootstrap

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

驗證（描述）：

kubectl describe clusterrolebinding cluster-system-anonymous

kubectl describe clusterrolebinding kubelet-bootstrap

2.5.9.2.2 創建kubelet配置文件

cat > kubelet.json << "EOF"
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.1.92",
  "port": 10250,
  "readOnlyPort": 10255,
  "cgroupDriver": "systemd",                    
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.96.0.2"]
}
EOF

2.5.9.2.3 創建kubelet服務啟動管理文件

注：

After 　　代表誰在前啟動。
Requires 代表需要哪個服務。
次此使用的容器運行時是：containerd，如果使用docker，兩處都填寫docker.service。
registry.aliyuncs.com/google_containers/pause:3.2或者3.6 都可以。

cat > kubelet.service << "EOF"
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet.json \
  --container-runtime=remote \
  --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
  --rotate-certificates \
  --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 \
  --root-dir=/etc/cni/net.d \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

查看生成的文件：

cd /data/k8s-work/
ls
kubelet-bootstrap.kubeconfig                                  
kubelet.json                          
kubelet.service

2.5.9.2.4 同步文件到集群節點

拷貝文件到指定目錄:

cp kubelet-bootstrap.kubeconfig /etc/kubernetes/
cp kubelet.json /etc/kubernetes/
cp kubelet.service /usr/lib/systemd/system/

worker節點創建目錄（master之前已經創建過）:

mkdir -p /etc/kubernetes/ssl

同步文件到其他三個節點：

for i in  k8s-master2 k8s-master3 k8s-worker1;do scp kubelet-bootstrap.kubeconfig kubelet.json $i:/etc/kubernetes/;done

for i in  k8s-master2 k8s-master3 k8s-worker1;do scp ca.pem $i:/etc/kubernetes/ssl/;done

for i in k8s-master2 k8s-master3 k8s-worker1;do scp kubelet.service $i:/usr/lib/systemd/system/;done

說明：

kubelet.json中address需要修改為當前主機IP地址（其他三臺：msater2、master3、worker1）。

2.5.9.2.5 創建目錄及啟動服務

注：所有節點創建目錄，否則kubelet 無法啟動！

mkdir -p /var/lib/kubelet
mkdir -p /var/log/kubernetes

啟動服務：

systemctl daemon-reload
systemctl enable --now kubelet

systemctl status kubelet

查看集群節點狀態：

[root@k8s-master1 k8s-work]# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
k8s-master1   Ready    <none>   13m   v1.25.5
k8s-master2   Ready    <none>   18s   v1.25.5
k8s-master3   Ready    <none>   12s   v1.25.5
k8s-worker1   Ready    <none>   10s   v1.25.5

查看bootstrap請求：

[root@k8s-master1 k8s-work]# kubectl get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
csr-24h9v   2m16s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
csr-rhpgr   15m     kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
csr-rq2xx   2m8s    kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
csr-xsrd7   2m10s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued

說明：

確認kubelet服務啟動成功后，接著到master上Approve一下bootstrap請求。

2.5.9.3 部署kube-proxy

2.5.9.3.1 創建kube-proxy證書請求文件

cat > kube-proxy-csr.json << "EOF"
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ]
}
EOF

2.5.9.3.2 生成證書

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

查看生成的證書：

ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

2.5.9.3.3 創建kubeconfig文件

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.100:6443 --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

2.5.9.3.4 創建服務配置文件

cat > kube-proxy.yaml << "EOF"
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.1.92
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 10.244.0.0/16
healthzBindAddress: 192.168.1.92:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.1.92:10249
mode: "ipvs"
EOF

2.5.9.3.5 創建服務啟動管理文件

cat >  kube-proxy.service << "EOF"
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.yaml \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

2.5.9.3.6 同步文件到集群工作節點主機

拷貝文件至master1指定目錄：

cp kube-proxy*.pem /etc/kubernetes/ssl/
cp kube-proxy.kubeconfig kube-proxy.yaml /etc/kubernetes/
cp kube-proxy.service /usr/lib/systemd/system/

同步文件到其他三個節點（master2、master3、worker1）：

for i in k8s-master2 k8s-master3 k8s-worker1;do scp kube-proxy.kubeconfig kube-proxy.yaml $i:/etc/kubernetes/;done
for i in k8s-master2 k8s-master3 k8s-worker1;do scp  kube-proxy.service $i:/usr/lib/systemd/system/;done

說明：

修改kube-proxy.yaml中IP地址為當前主機IP.

2.5.9.3.7 服務啟動

創建目錄（四臺主機master1、2、3和worker1）：

mkdir -p /var/lib/kube-proxy

啟動服務：

systemctl daemon-reload
systemctl enable --now kube-proxy

systemctl status kube-proxy

2.5.10 網絡組件部署 Calico

2.5.10.1 下載

curl https://projectcalico.docs.tigera.io/archive/v3.23/manifests/calico.yaml -O

2.5.10.2 修改文件

4470 - name: CALICO_IPV4POOL_CIDR
4471   value: "10.244.0.0/16"

2.5.10.3 應用文件

kubectl apply -f calico.yaml

2.5.10.4 驗證應用結果

[root@k8s-master1 k8s-work]#  kubectl get pods -A -o wide
NAMESPACE     NAME                                      READY   STATUS    RESTARTS      AGE    IP              NODE          NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-d8b9b6478-g26fc   1/1     Running   2 (31s ago)   3m2s   10.244.194.64   k8s-worker1   <none>           <none>
kube-system   calico-node-7hx5p                         1/1     Running   0             3m2s   192.168.1.93    k8s-master2   <none>           <none>
kube-system   calico-node-crd6m                         1/1     Running   0             3m2s   192.168.1.96    k8s-worker1   <none>           <none>
kube-system   calico-node-jnvs7                         1/1     Running   0             3m2s   192.168.1.94    k8s-master3   <none>           <none>
kube-system   calico-node-k8qth                         1/1     Running   0             3m2s   192.168.1.92    k8s-master1   <none>           <none>

集群狀態：

[root@k8s-master1 k8s-work]# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
k8s-master1   Ready    <none>   84m   v1.25.5
k8s-master2   Ready    <none>   71m   v1.25.5
k8s-master3   Ready    <none>   70m   v1.25.5
k8s-worker1   Ready    <none>   70m   v1.25.5

2.5.10 部署CoreDNS

cat >  coredns.yaml << "EOF"
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
rules:
  - apiGroups:
    - ""
    resources:
    - endpoints
    - services
    - pods
    - namespaces
    verbs:
    - list
    - watch
  - apiGroups:
    - discovery.k8s.io
    resources:
    - endpointslices
    verbs:
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local  in-addr.arpa ip6.arpa {
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  # replicas: not specified here:
  # 1. Default is 1.
  # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - weight: 100
             podAffinityTerm:
               labelSelector:
                 matchExpressions:
                   - key: k8s-app
                     operator: In
                     values: ["kube-dns"]
               topologyKey: kubernetes.io/hostname
      containers:
      - name: coredns
        image: coredns/coredns:1.8.4
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP
 
EOF

安裝：

kubectl apply -f coredns.yaml

查看狀態：

[root@k8s-master1 k8s-work]# kubectl get pods -n kube-system -o wide
NAME                                      READY   STATUS    RESTARTS      AGE   IP               NODE          NOMINATED NODE   READINESS GATES
calico-kube-controllers-d8b9b6478-g26fc   1/1     Running   2 (10m ago)   13m   10.244.194.64    k8s-worker1   <none>           <none>
calico-node-7hx5p                         1/1     Running   0             13m   192.168.1.93     k8s-master2   <none>           <none>
calico-node-crd6m                         1/1     Running   0             13m   192.168.1.96     k8s-worker1   <none>           <none>
calico-node-jnvs7                         1/1     Running   0             13m   192.168.1.94     k8s-master3   <none>           <none>
calico-node-k8qth                         1/1     Running   0             13m   192.168.1.92     k8s-master1   <none>           <none>
coredns-564fd8c776-h7f4q                  1/1     Running   0             29s   10.244.159.129   k8s-master1   <none>           <none>

驗證測試：

dig -t a www.baidu.com @10.96.0.2

2.5.11 部署應用驗證

cat >  nginx.yaml  << "EOF"
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-web
spec:
  replicas: 2
  selector:
    name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.19.6
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service-nodeport
spec:
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30001
      protocol: TCP
  type: NodePort
  selector:
    name: nginx
EOF

（1）安裝部署：

kubectl apply -f nginx.yaml

（2）查看pod狀態

[root@k8s-master1 k8s-work]# kubectl get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE     IP               NODE          NOMINATED NODE   READINESS GATES
nginx-web-cxgtm   1/1     Running   0          2m16s   10.244.135.193   k8s-master3   <none>           <none>
nginx-web-nd8hg   1/1     Running   0          2m16s   10.244.224.1     k8s-master2   <none>           <none>

（3）查看所有default下內容：

[root@k8s-master1 k8s-work]# kubectl get all
NAME                  READY   STATUS    RESTARTS   AGE
pod/nginx-web-cxgtm   1/1     Running   0          11m
pod/nginx-web-nd8hg   1/1     Running   0          11m

NAME                              DESIRED   CURRENT   READY   AGE
replicationcontroller/nginx-web   2         2         2       11m

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/kubernetes               ClusterIP   10.96.0.1      <none>        443/TCP        30h
service/nginx-service-nodeport   NodePort    10.96.216.41   <none>        80:30001/TCP   11m

（4）查看節點端口：

ss -anput | grep "30001"

訪問測試：

至此，集群部署完畢！

posted @ 2022-12-16 18:17 西瓜君~ 閱讀(771) 評論(0) 收藏舉報

刷新頁面返回頂部

西瓜君~

Centos 7.9 基于二進制文件部署kubernetes v1.25.5集群

簡述

一、集群環境準備

1.1 主機規劃

1.2 軟件版本

1.3 網絡分配

二、集群部署

2.1主機準備

2.1.1 主機名設置

2.1.2 主機與IP地址解析

2.1.3 主機安全設置

2.1.3.1 關閉防火墻

2.1.3.2 關閉selinux

2.1.4 交換分區設置

2.1.5 主機系統時間同步

2.1.6 主機系統優化

2.1.7 ipvs管理工具安裝及模塊加載

2.1.8 加載containerd相關內核模塊

2.1.8.1 臨時加載模塊

2.1.8.2 永久性加載模塊

2.1.8.3 設置為開機啟動

2.1.9 Linux內核升級

2.1.10 Linux內核優化

2.1.11 其它工具安裝(選裝)

2.2 負載均衡器準備

2.2.1 安裝haproxy與keepalived

2.2.2 HAProxy配置

2.2.3 KeepAlived

2.2.4 健康檢查腳本

2.2.5 啟動服務并驗證

2.3 配置免密登錄

2.3.1 生成密鑰：

2.3.2 拷貝至其他服務器

2.3.3 測試免密登錄

2.4 創建CA及etcd證書

2.4.1 創建工作目錄

2.4.2 獲取cfssl工具

2.4.3 創建CA證書

2.4.3.1 配置ca證書請求文件

2.4.3.2 創建ca證書

2.4.3.3 生成配置ca證書策略

2.4.4 創建etcd證書

2.4.4.1 配置etcd請求文件

2.4.4.2 生成etcd證書

2.4.4.3 查看生成的etcd證書文件

2.4.5 部署etcd集群

2.4.5.1 下載etcd軟件包

2.4.5.2 安裝etcd軟件

2.4.5.3 分發etcd軟件

2.4.5.4 創建配置文件

2.4.5.5 創建服務配置文件

2.4.5.6 同步etcd配置到集群其它master節點

2.4.5.7 啟動etcd集群

2.4.5.8 驗證集群狀態

2.5 Kubernetes集群部署

2.5.1 Kubernetes軟件包下載

2.5.2 Kubernetes軟件包安裝

2.5.3 Kubernetes軟件分發

2.5.4 在集群節點上創建目錄

2.5.5 部署api-server

2.5.5.1 創建apiserver證書請求文件

2.5.5.2 生成apiserver證書及token文件

2.5.5.3 創建apiserver服務配置文件

2.5.5.4 創建apiserver服務管理配置文件

2.5.5.5 同步文件到集群master節點

2.5.5.6 啟動apiserver服務

2.5.5.7 測試

2.5.6 部署kubectl

2.5.6.1 創建kubectl證書請求文件

2.5.6.2 生成證書文件

2.5.6.3 復制文件到指定目錄

2.5.6.4 生成kubeconfig配置文件

2.5.6.5 準備kubectl配置文件并進行角色綁定

2.5.6.6 查看集群狀態

2.5.6.7 同步kubectl配置文件到集群其它master節點

2.5.6.8 配置kubectl命令補全(可選)

2.5.7 部署kube-controller-manager

2.5.7.1 創建kube-controller-manager證書請求文件

2.5.7.2 創建kube-controller-manager證書文件

二、集群部署