k8s work node status NotReady

問題背景：

Work Node 節點執行 join后，狀態NotReady.

kubectl get node
NAME          STATUS     ROLES           AGE     VERSION
k8s-master2   Ready      control-plane   2d21h   v1.29.0
k8s-node1     NotReady   <none>          2d21h   v1.29.0

然后查看pod可以看到192.168.56.11的flannel pod沒有啟動起來。

kubectl get pod -A -owide
NAMESPACE      NAME                                  READY   STATUS              RESTARTS   AGE     IP               NODE          NOMINATED NODE   READINESS GATES
kube-flannel   kube-flannel-ds-j2q5w                 1/1     Running             2          2d21h   192.168.56.100   k8s-master2   <none>           <none>
kube-flannel   kube-flannel-ds-n5h7t                 0/1     Init:0/2            0          14h     192.168.56.11    k8s-node1     <none>           <none>
kube-system    coredns-857d9ff4c9-9vw86              1/1     Running             2          2d21h   10.244.0.7       k8s-master2   <none>           <none>
kube-system    coredns-857d9ff4c9-fxglw              1/1     Running             2          2d21h   10.244.0.6       k8s-master2   <none>           <none>
kube-system    etcd-k8s-master2                      1/1     Running             6          2d21h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    kube-apiserver-k8s-master2            1/1     Running             8          2d21h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    kube-controller-manager-k8s-master2   1/1     Running             2          2d21h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    kube-proxy-p528v                      0/1     ContainerCreating   0          14h     192.168.56.11    k8s-node1     <none>           <none>
kube-system    kube-proxy-vckjv                      1/1     Running             2          2d21h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    kube-scheduler-k8s-master2            1/1     Running             7          2d21h   192.168.56.100   k8s-master2   <none>           <none>

接下來再查看node節點，kubelet的狀態

這里有這樣的錯誤sandbox config for pod \\\"kube-flannel-ds-n5h7t_kube-flannel(7410979a-2b8c-4bae-852d-49ee657fe9cb)\\\": open /run/systemd/resolve/resolv.conf: no such file or directory\""

這里pod ‘kube-flannel-ds-n5h7t_kube-flannel’ 啟動失敗了。所以要做的就是解決這個問題。

解決方案

sudo apt update
sudo apt install systemd-resolved
sudo systemctl start systemd-resolved.service
sudo systemctl enable systemd-resolved.service
Debian 相關的技術文檔
https://manpages.debian.org/testing/systemd/systemd-resolved.service.8.en.html
https://packages.debian.org/unstable/systemd-resolved

之后異常改成了這樣

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Mon 2024-01-01 06:26:04 EST; 14h ago
       Docs: https://kubernetes.io/docs/
   Main PID: 2298 (kubelet)
      Tasks: 11 (limit: 2307)
     Memory: 36.7M
        CPU: 34.151s
     CGroup: /system.slice/kubelet.service
             └─2298 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --contain>

Jan 01 20:46:24 k8s-node1 kubelet[2298]: E0101 20:46:24.591540    2298 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady >
Jan 01 20:46:25 k8s-node1 kubelet[2298]: E0101 20:46:25.955155    2298 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to ge>
Jan 01 20:46:25 k8s-node1 kubelet[2298]: E0101 20:46:25.955477    2298 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get san>
Jan 01 20:46:25 k8s-node1 kubelet[2298]: E0101 20:46:25.955577    2298 kuberuntime_manager.go:1172] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sa>
lines 1-17

繼續檢查containerd問題，發先沒有/etc/containerd/config.toml文件，于時執行containerd config default > /etc/containerd/config.toml生產一份默認的配置文件。并重啟containerd和kubelet。

這里，最好修改config.toml的sandbox鏡像倉庫地址

    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"

sudo systemctl restart containerd && sudo systemctl restart kubelet

最后的運行效果

master@k8s-master2:~$ kubectl get pods -A -owide
NAMESPACE      NAME                                  READY   STATUS    RESTARTS      AGE     IP               NODE          NOMINATED NODE   READINESS GATES
kube-flannel   kube-flannel-ds-flfgs                 1/1     Running   3 (84s ago)   50m     192.168.56.11    k8s-node1     <none>           <none>
kube-flannel   kube-flannel-ds-j2q5w                 1/1     Running   2             3d18h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    coredns-857d9ff4c9-9vw86              1/1     Running   2             3d18h   10.244.0.7       k8s-master2   <none>           <none>
kube-system    coredns-857d9ff4c9-fxglw              1/1     Running   2             3d18h   10.244.0.6       k8s-master2   <none>           <none>
kube-system    etcd-k8s-master2                      1/1     Running   6             3d18h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    kube-apiserver-k8s-master2            1/1     Running   8             3d18h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    kube-controller-manager-k8s-master2   1/1     Running   2             3d18h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    kube-proxy-p528v                      1/1     Running   4 (83s ago)   35h     192.168.56.11    k8s-node1     <none>           <none>
kube-system    kube-proxy-vckjv                      1/1     Running   2             3d18h   192.168.56.100   k8s-master2   <none>           <none>
kube-system    kube-scheduler-k8s-master2            1/1     Running   7             3d18h   192.168.56.100   k8s-master2   <none>           <none>
master@k8s-master2:~$ kubectl get node
NAME          STATUS   ROLES           AGE     VERSION
k8s-master2   Ready    control-plane   3d18h   v1.29.0
k8s-node1     Ready    <none>          3d18h   v1.29.0

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Mon 2024-01-01 21:51:40 EST; 6min ago
       Docs: https://kubernetes.io/docs/
   Main PID: 1676 (kubelet)
      Tasks: 10 (limit: 2307)
     Memory: 34.1M
        CPU: 3.454s
     CGroup: /system.slice/kubelet.service
             └─1676 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --c>
lines 1-12

posted on 2024-01-02 11:09 iTrice 閱讀(94) 評論(0) 收藏舉報

刷新頁面返回頂部

k8s work node status NotReady

問題背景：

解決方案

公告