1 docker安裝部署

1.1 docker介紹

首先 Docker 是一個(gè)在 2013 年開(kāi)源的應(yīng)用程序并且是一個(gè)基于 go 語(yǔ)言編寫是一個(gè)開(kāi)源的 PAAS 服務(wù)(Platform as a Service，平臺(tái)即服務(wù)的縮寫)，go 語(yǔ)言是由google 開(kāi)發(fā)，docker 公司最早叫 dotCloud 后由于 Docker 開(kāi)源后大受歡迎就將公司改名為 Docker Inc，總部位于美國(guó)加州的舊金山，Docker 是基于 linux 內(nèi)核實(shí)現(xiàn)，Docker 最早采用 LXC 技術(shù)(LinuX Container 的簡(jiǎn)寫，LXC 是 Linux 原生支持的容器技術(shù)，可以提供輕量級(jí)的虛擬化，可以說(shuō) docker 就是基于 LXC 發(fā)展起來(lái)的(0.1.5 (2013-04-17)，提供 LXC 的高級(jí)封裝，發(fā)展標(biāo)準(zhǔn)的配置方法)，而虛擬化 技術(shù) KVM(Kernel-based Virtual Machine) 基于模塊實(shí)現(xiàn)，Docker 后改為自己研發(fā)并開(kāi)源的 runc 技術(shù)運(yùn)行容器(1.11.0 (2016-04-13)。

1.1.1 Linux Namespace 技術(shù)

namespace 是 Linux 系統(tǒng)的底層概念，在內(nèi)核層實(shí)現(xiàn)，即有一些不同類型的命名空間被部署在核內(nèi)，各個(gè) docker 容器運(yùn)行在同一個(gè) docker 主進(jìn)程并且共用同一個(gè)宿主機(jī)系統(tǒng)內(nèi)核，各 docker 容器運(yùn)行在宿主機(jī)的用戶空間，每個(gè)容器都要有類似于虛擬機(jī)一樣的相互隔離的運(yùn)行空間，但是容器技術(shù)是在一個(gè)進(jìn)程內(nèi)實(shí)現(xiàn)運(yùn)行指定服務(wù)的運(yùn)行環(huán)境，并且還可以保護(hù)宿主機(jī)內(nèi)核不受其他進(jìn)程的干擾和影響，如文件系統(tǒng)空間、網(wǎng)絡(luò)空間、進(jìn)程空間等

隔離類別	功能	系統(tǒng)調(diào)用參數(shù)	內(nèi)核版本
MNT Namespace	提供磁盤掛載點(diǎn)和文件系統(tǒng)的隔離能力	CLONE_NEWNS	Linux2.4.19
IPC Namespace(Inter-Process Communication)	提供進(jìn)程間通信的隔離能力	CLONE_NEWIPC	Linux2.6.19
UTS Namespace(UNIX Timesharing System)	提供主機(jī)名隔離能力	CLONE_NEWUTS	Linux2.6.19
PID Namespace(Process Identification)	提供進(jìn)程隔離能力	CLONE_NEWPID	Linux2.6.24
Net Namespace(network)	提供網(wǎng)絡(luò)隔離能力	CLONE_NEWNET	Linux2.6.29
User Namespace(user)	提供用戶隔離能力	CLONE_NEWUSER	Linux3.8

#MNT Namespace 
每個(gè)容器有獨(dú)立的根文件系統(tǒng)和獨(dú)立的用戶空間，以實(shí)現(xiàn)在容器里面啟動(dòng)服務(wù)并且使用容器的運(yùn)行環(huán)境。
即一個(gè)宿主機(jī)是ubuntu的服務(wù)器，可以在里面啟動(dòng)一個(gè)centos運(yùn)行環(huán)境的容器并且在容器里面啟動(dòng)一個(gè)Nginx服務(wù)，此 Nginx運(yùn)行時(shí)使用的運(yùn)行環(huán)境就是centos系統(tǒng)目錄的運(yùn)行環(huán)境，但是在容器里面是不能訪問(wèn)宿主機(jī)的資源，宿主機(jī)是使用了 chroot技術(shù)把容器鎖定到一個(gè)指定的運(yùn)行目錄里面
例如 #：containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/`容器ID` -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc


#IPC Namespace
一個(gè)容器內(nèi)的進(jìn)程間通信，允許一個(gè)容器內(nèi)的不同進(jìn)程的(內(nèi)存、緩存等)數(shù)據(jù)訪問(wèn)，但是不能夸容器訪問(wèn)其他容器的數(shù)據(jù)。

#UTS Namespace
UTS namespace（UNIX Timesharing System 包含了運(yùn)行內(nèi)核的名稱、版本、底層體 系結(jié)構(gòu)類型等信息）用于系統(tǒng)標(biāo)識(shí)，其中包含hostname和域名domainname，它使得一個(gè)容器擁有屬于自己hostname標(biāo)識(shí)，這個(gè)主機(jī)名標(biāo)識(shí)獨(dú)立于宿主機(jī)系統(tǒng)和其上的其他容器。
root@harbor:~# docker exec -it 2b9314b71794 bash
nginx [ / ]$ ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  proc  root  run  sbin  srv  sys  tmp  usr  var
nginx [ / ]$ cat /etc/issue
Welcome to Photon 4.0 (\m) - Kernel \r (\l)
nginx [ / ]$ uname -a  #宿主機(jī)的內(nèi)核
Linux 2b9314b71794 4.15.0-152-generic #159-Ubuntu SMP Fri Jul 16 16:28:09 UTC 2021 x86_64
nginx [ / ]$ hostname  #自己的hostname
2b9314b71794

#PID namespace
Linux 系統(tǒng)中，有一個(gè) PID 為 1 的進(jìn)程(init/systemd)是其他所有進(jìn)程的父進(jìn)程，那么在每個(gè)容器內(nèi)也要有一個(gè)父進(jìn)程來(lái)管理其下屬的子進(jìn)程，那么多個(gè)容器的進(jìn)程通PID namespace進(jìn)程隔離
root@harbor:~# docker exec -it 2b9314b71794 bash  #查看當(dāng)前父進(jìn)程nginx: master process nginx -g
nginx [ / ]$ ps -ef
UID             PID   PPID C STIME TTY          TIME CMD
nginx             1      0 0 48:05 ?        00:00:00 nginx: master process nginx -g daemon off;
nginx             6      1 0 48:05 ?        00:00:00 nginx: worker process 
nginx             7      1 0 48:05 ?        00:00:00 nginx: worker process
nginx           346      0 0 10:09 pts/0    00:00:00 bash
nginx           445    346 0 15:46 pts/0    00:00:00 ps -ef
root@harbor:~# ps -ef |grep docker #找容器id對(duì)應(yīng)的宿主機(jī)的目錄
root        379      1  0 07:47 ?        00:00:04 /opt/kube/bin/dockerd
root        549    379  0 07:47 ?        00:00:02 containerd --config /var/run/docker/containerd/containerd.toml --log-level warn
root       1152    379  0 07:48 ?        00:00:00 /opt/kube/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.3 -container-port 8080 #docker-proxy 負(fù)責(zé)容器訪問(wèn)
root       1164    379  0 07:48 ?        00:00:00 /opt/kube/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 1514 -container-ip 172.18.0.9 -container-port 10514
root       1171    549  0 07:48 ?        00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
root       1192    549  0 07:48 ?        00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/89e6038bf99fb14a7c586b1317b8eb4e08ec2dcbff4d5055d596a3ac80cfed9c -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
root       1202    549  0 07:48 ?        00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/5dbda7bfb4f52dc6abd5d16f9f311585426c2d091eda339c3c19d1efdf7896fa -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
root@harbor:~# ps -ef |grep 1171
root       1171    549  0 07:48 ?        00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
10000      1411   1171  0 07:48 ?        00:00:00 nginx: master process nginx -g daemon off;
root      33661  30705  0 08:20 pts/1    00:00:00 grep --color=auto 1171
root@harbor:~# ps -ef |grep 1411
10000      1411   1171  0 07:48 ?        00:00:00 nginx: master process nginx -g daemon off;
10000      1610   1411  0 07:48 ?        00:00:00 nginx: worker process
10000      1611   1411  0 07:48 ?        00:00:00 nginx: worker process

#Net Namespace
每一個(gè)容器都類似于虛擬機(jī)一樣有自己的網(wǎng)卡、監(jiān)聽(tīng)端口、TCP/IP 協(xié)議棧等， Docker使用network namespace啟動(dòng)一個(gè)`vethX`接口，這樣你的容器將擁有它自己的橋接ip地址，通常是docker0，而docker0實(shí)質(zhì)就是Linux的虛擬網(wǎng)橋,網(wǎng)橋 是在OSI七層模型的數(shù)據(jù)鏈路層的網(wǎng)絡(luò)設(shè)備，通過(guò)mac地址對(duì)網(wǎng)絡(luò)進(jìn)行劃分，并且在不同網(wǎng)絡(luò)直接傳遞數(shù)據(jù)。

#User Namespace
User Namespace允許在各個(gè)宿主機(jī)的各個(gè)容器空間內(nèi)創(chuàng)建相同的用戶名以及相同的用戶UID和GID，只是會(huì)把用戶的作用范圍限制在每個(gè)容器內(nèi)，即A容器和B容器可以有相同的用戶名稱/ID的賬戶，但是此用戶的有效范圍僅是當(dāng)前容器內(nèi)，不能訪問(wèn)另外一個(gè)容器內(nèi)的文件系統(tǒng)，即相互隔離。

1.1.2 Linux control groups

在一個(gè)容器，如果不對(duì)其做任何資源限制，則宿主機(jī)會(huì)允許其占用無(wú)限大的內(nèi)存空間，有時(shí)候會(huì)因?yàn)榇abug程序會(huì)一直申請(qǐng)內(nèi)存，直到把宿主機(jī)內(nèi)存占完，為了避免此類的問(wèn)題出現(xiàn)，宿主機(jī)有必要對(duì)容器進(jìn)行資源分配限制，比如CPU、內(nèi)存等，Linux Cgroups 的全稱是 Linux Control Groups，它最主要的作用，就是限制一個(gè)進(jìn)程組能夠使用的資源上限，包括 CPU、內(nèi)存、磁盤、網(wǎng)絡(luò)帶寬等等。此外，還能夠?qū)M(jìn)程進(jìn)行優(yōu)先級(jí)設(shè)置，以及將進(jìn)程掛起和恢復(fù)等操作

1.1.2.1 查看系統(tǒng)cgroups

root@harbor:~/harbor# ll /sys/fs/cgroup/
total 0
drwxr-xr-x 15 root root 380 Sep  5 07:47 ./
drwxr-xr-x  9 root root   0 Sep  5 07:47 ../
dr-xr-xr-x  5 root root   0 Sep  5 07:47 blkio/  #blkio：塊設(shè)備 IO 限制。
lrwxrwxrwx  1 root root  11 Sep  5 07:47 cpu -> cpu,cpuacct/  #使用調(diào)度程序?yàn)?cgroup 任務(wù)提供 cpu 的訪問(wèn)
dr-xr-xr-x  5 root root   0 Sep  5 07:47 cpu,cpuacct/  
lrwxrwxrwx  1 root root  11 Sep  5 07:47 cpuacct -> cpu,cpuacct/ #產(chǎn)生 cgroup 任務(wù)的 cpu 資源報(bào)告。
dr-xr-xr-x  3 root root   0 Sep  5 07:47 cpuset/ #如果是多核心的 cpu，這個(gè)子系統(tǒng)會(huì)為 cgroup 任務(wù)分配單獨(dú)的 cpu 和 內(nèi)存。
dr-xr-xr-x  5 root root   0 Sep  5 07:47 devices/  #允許或拒絕 cgroup 任務(wù)對(duì)設(shè)備的訪問(wèn)。
dr-xr-xr-x  3 root root   0 Sep  5 07:47 freezer/  #暫停和恢復(fù) cgroup 任務(wù)。
dr-xr-xr-x  3 root root   0 Sep  5 07:47 hugetlb/
dr-xr-xr-x  5 root root   0 Sep  5 07:47 memory/ #設(shè)置每個(gè) cgroup 的內(nèi)存限制以及產(chǎn)生內(nèi)存資源報(bào)告。
lrwxrwxrwx  1 root root  16 Sep  5 07:47 net_cls -> net_cls,net_prio/ #標(biāo)記每個(gè)網(wǎng)絡(luò)包以供 cgroup 方便使用。
dr-xr-xr-x  3 root root   0 Sep  5 07:47 net_cls,net_prio/
lrwxrwxrwx  1 root root  16 Sep  5 07:47 net_prio -> net_cls,net_prio/
dr-xr-xr-x  3 root root   0 Sep  5 07:47 perf_event/  #增加了對(duì)每 group 的監(jiān)測(cè)跟蹤的能力
dr-xr-xr-x  5 root root   0 Sep  5 07:47 pids/
dr-xr-xr-x  2 root root   0 Sep  5 07:47 rdma/
dr-xr-xr-x  6 root root   0 Sep  5 07:47 systemd/
dr-xr-xr-x  5 root root   0 Sep  5 07:47 unified/

1.1.2.2 查看容器的資源限制

目前 docker 已經(jīng)幾乎支持了所有的 cgroups 資源，可以限制容器對(duì)包括 network，device，cpu 和 memory 在內(nèi)的資源的使用

默認(rèn)情況下，Docker 啟動(dòng)一個(gè)容器后，會(huì)在 /sys/fs/cgroup 目錄下的各個(gè)資源目錄下生成以容器 ID 為名字的目錄（group）

/sys/fs/cgroup/cpu/docker/03dd196f415276375f754d51ce29b418b170bd92d88c5e420d6901c32f93dc14

root@harbor:/var/lib/docker# cd /sys/fs/cgroup/
root@harbor:/sys/fs/cgroup# ls
blkio  cpu,cpuacct  cpuset   freezer  memory   net_cls,net_prio  perf_event  rdma     unified
cpu    cpuacct      devices  hugetlb  net_cls  net_prio          pids        systemd
root@harbor:/sys/fs/cgroup# find ./* -iname 2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./blkio/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./cpu,cpuacct/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./cpuset/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./devices/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./freezer/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./hugetlb/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./memory/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./net_cls,net_prio/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./perf_event/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./pids/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./systemd/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c

1.1.2.3 docker中run 命令中 cgroups 相關(guān)命令

block IO:
      --blkio-weight value          Block IO (relative weight), between 10 and 1000
      --blkio-weight-device value   Block IO weight (relative device weight) (default [])
      --cgroup-parent string        Optional parent cgroup for the container
CPU:
      --cpu-percent int             CPU percent (Windows only)
      --cpu-period int              Limit CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int               Limit CPU CFS (Completely Fair Scheduler) quota
  -c, --cpu-shares int              CPU shares (relative weight)
      --cpuset-cpus string          CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string          MEMs in which to allow execution (0-3, 0,1)
Device:    
      --device value                Add a host device to the container (default [])
      --device-read-bps value       Limit read rate (bytes per second) from a device (default [])
      --device-read-iops value      Limit read rate (IO per second) from a device (default [])
      --device-write-bps value      Limit write rate (bytes per second) to a device (default [])
      --device-write-iops value     Limit write rate (IO per second) to a device (default [])
Memory:      
      --kernel-memory string        Kernel memory limit
  -m, --memory string               Memory limit
      --memory-reservation string   Memory soft limit
      --memory-swap string          Swap limit equal to memory plus swap: '-1' to enable unlimited swap
      --memory-swappiness int       Tune container memory swappiness (0 to 100) (default -1)

1.1.3 cgroups驗(yàn)證

1.1.3.1 創(chuàng)建容器的CPU權(quán)重控制

默認(rèn)情況下，每個(gè)docker容器的cpu份額都是1024，單獨(dú)一個(gè)容器的份額是沒(méi)有意義的，只有在同時(shí)運(yùn)行多個(gè)容器時(shí)，容器cpu的加權(quán)效果才能體現(xiàn)出現(xiàn)。

例如，兩個(gè)容器A、B的cpu份額分別為1000和500，在cpu進(jìn)行時(shí)間片分配的時(shí)候，容器A比容器B多一倍的機(jī)會(huì)獲得cpu的時(shí)間片，但是分配的結(jié)果取決于當(dāng)時(shí)主機(jī)和其他容器的運(yùn)行狀態(tài)，實(shí)際上也無(wú)法保證容器A一定能夠獲得cpu的時(shí)間片。比如容器A的進(jìn)程一直是空閑的，那么容器B是可以獲取比容器A更多的cpu時(shí)間片的，極端情況下，例如主機(jī)上只運(yùn)行的一個(gè)容器，即使它的cpu份額只有50，它也可以獨(dú)占整個(gè)主機(jī)的cpu資源

cgroups只在容器分配的資源緊缺時(shí)，即需要對(duì)容器使用的資源進(jìn)行限制時(shí)，才會(huì)生效。因此，無(wú)法單純的根據(jù)某個(gè)容器的份額的cpu份額來(lái)確定有多少cpu資源分配給它，可以通過(guò)cpu share參數(shù)可以設(shè)置容器使用cpu的優(yōu)先級(jí)，比如啟動(dòng)了兩個(gè)容器及運(yùn)行查看cpu的cpu的使用百分比

創(chuàng)建兩個(gè)容器，分別制定不同的權(quán)重比

#docker run -itd --name cpu1024 --cpu-shares 1024 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -c 10

#docker run -itd --name cpu512 --cpu-shares 512 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -c 10

查看%cpu的比例

root@harbor2:~# docker exec -it cpu1024 bash
top - 02:27:46 up  2:39,  0 users,  load average: 19.86, 13.08, 5.86
Tasks:  13 total,  11 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu0  : 99.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  4015804 total,  2211948 free,   261200 used,  1542656 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used.  3512012 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                            
     7 root      20   0    7324    100      0 R  19.9  0.0   0:51.90 stress                             
     8 root      20   0    7324    100      0 R  19.9  0.0   0:51.87 stress                             
     9 root      20   0    7324    100      0 R  19.9  0.0   0:51.84 stress                             
    10 root      20   0    7324    100      0 R  19.9  0.0   0:51.86 stress                             
    11 root      20   0    7324    100      0 R  19.9  0.0   0:51.86 stress                             
    12 root      20   0    7324    100      0 R  19.9  0.0   0:51.90 stress  
    
------------------------------------------------
root@harbor2:~# docker exec -it cpu512 bash
top - 02:28:40 up  2:40,  0 users,  load average: 19.94, 14.25, 6.67
Tasks:  13 total,  11 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  4015804 total,  2210800 free,   262280 used,  1542724 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used.  3511016 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                            
     6 root      20   0    7324     96      0 R  10.0  0.0   1:08.04 stress                             
     7 root      20   0    7324     96      0 R  10.0  0.0   1:08.11 stress                             
     8 root      20   0    7324     96      0 R  10.0  0.0   1:08.02 stress                             
     9 root      20   0    7324     96      0 R  10.0  0.0   1:08.40 stress                             
    10 root      20   0    7324     96      0 R  10.0  0.0   1:07.82 stress                             
    11 root      20   0    7324     96      0 R  10.0  0.0   1:07.89 stress 

分別進(jìn)入cpu512和cpu1024之后可以看到，%cpu的比例差不多是1：2，符合我們?cè)O(shè)置的–cpu-shares參數(shù)

1.1.3.2 cpu core控制

對(duì)于多核cpu的服務(wù)器，docker還可以控制容器運(yùn)行使用那些cpu內(nèi)核，以及使用–cpuset-cpus參數(shù)，這對(duì)于具有多cpu服務(wù)器尤其有用，可以對(duì)需要高性能計(jì)算的容器進(jìn)行性能最優(yōu)的配置

root@harbor2:~# docker run -itd --name cpu1 --cpuset-cpus 0-1 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1
326cacc9423fefe5f942441f449f8f22c9319ce59d28c18ca6fbb4651d8263c5
root@harbor2:~# docker exec -it cpu1 bash
[root@326cacc9423f /]# cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-1

//通過(guò)下列指令可以看到容器中進(jìn)程與cpu內(nèi)核的綁定關(guān)系，達(dá)到綁定cpu內(nèi)核的目的
root@harbor2:~# docker exec -it cpu1 taskset -c -p 1
pid 1's current affinity list: 0,1    #容器內(nèi)部的第一個(gè)進(jìn)程號(hào)pid為1，被綁定到指定到的cpu上運(yùn)行

1.1.3.3 cpu配額控制參數(shù)的混合使用

docker run -itd --name cpu2 --cpuset-cpus 1 --cpu-shares 512 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -c 1

docker run -itd --name cpu3 --cpuset-cpus 3 --cpu-shares 1024 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -c 1

1.1.3.4 內(nèi)存配額

• 與操作系統(tǒng)類似，容器可使用的內(nèi)存包括兩部分：物理內(nèi)存和swap
  容器通過(guò) -m或–memory設(shè)置內(nèi)存的使用限額，例如：-m 300M；通過(guò)–memory-swap設(shè)置內(nèi)存+swap的使用限額
• 實(shí)例如下，允許容器最多使用200M的內(nèi)存和300M的swap

docker run -itd --name mem1 –m 300M --memory-swap=300M registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -m 3 --vm-bytes 300M
#會(huì)生成3個(gè)進(jìn)程，每個(gè)進(jìn)程占用300M內(nèi)存
root@harbor2:~# docker exec -it 0504806aa832 bash
#top之后會(huì)看到這300M使用的內(nèi)存在3個(gè)線程上來(lái)回飄

1.1.3.5 Block IO的限制

默認(rèn)情況下，所有容器能平等地讀寫磁盤，可以通過(guò)設(shè)置–blkio-weight參數(shù)來(lái)改變?nèi)萜鱞lock IO的優(yōu)先級(jí)

docker run -itd --name blokio1 --blkio-weight 600 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 

docker run -itd --name blokio2 --blkio-weight 300 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1

docker exec -it --name blokio2 bash
cat /sys/fs/cgroup/blkio/blkio.weight
300

1.1.3.6 bps和iops限制

bps是byte per second，每秒讀寫的數(shù)據(jù)量。iops是io per second, 每秒IO的次數(shù)。
可通過(guò)以下參數(shù)控制容器的bps和iops:

--device-read-bps：限制讀某個(gè)設(shè)備的bps.
--device-write-bps：限制寫某個(gè)設(shè)備的bps.
--device-read-iops：限制讀某個(gè)設(shè)備的iops.
--device-write-iops：限制寫某個(gè)設(shè)備的iops。

下面的示例是限制容器寫/dev/sda 的速率為5 MB/s

docker run -itd --name bps1 --device-write-bps /dev/sda:5MB registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1

docker exec -it bps1 bash
dd if=/dev/zero of=test2 bs=1M count=1024 oflag=direct

1.2 docker安裝和基礎(chǔ)命令

1.2.1 在線安裝

如果你過(guò)去安裝過(guò) docker，先刪掉:

sudo apt-get remove docker docker-engine docker.io

首先安裝依賴:

sudo apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common

根據(jù)你的發(fā)行版

信任 Docker 的 GPG 公鑰:

curl -fsSL https://download.docker.com/linux/[ubuntu|debian]/gpg | sudo apt-key add -
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

添加軟件倉(cāng)庫(kù):

echo 'deb [arch=amd64] https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/ubuntu bionic stable' >> /etc/apt/sources.list

最后安裝

sudo apt-get update
sudo apt-get install docker-ce

驗(yàn)證

root@harbor2:~# docker run -it --rm nginx bash
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
a330b6cecb98: Pull complete 
5ef80e6f29b5: Pull complete 
f699b0db74e3: Pull complete 
0f701a34c55e: Pull complete 
3229dce7b89c: Pull complete 
ddb78cb2d047: Pull complete 
Digest: sha256:a05b0cdd4fc1be3b224ba9662ebdf98fe44c09c0c9215b45f84344c12867002e
Status: Downloaded newer image for nginx:latest
root@d8840d0d3a84:/# exit 
exit

1.2.2 離線安裝

cat >> limits.conf << EOF
*             soft    core            unlimited
*             hard    core            unlimited
*	      soft    nproc           1000000
*             hard    nproc           1000000
*             soft    nofile          1000000
*             hard    nofile          1000000
*             soft    memlock         32000
*             hard    memlock         32000
*             soft    msgqueue        8192000
*             hard    msgqueue        8192000
EOF
#如果是非root用戶將*修改成用戶名稱再添加一次
cat >> sysctl.conf << EOF
net.ipv4.ip_forward=1
vm.max_map_count=262144
kernel.pid_max=4194303
fs.file-max=1000000
net.ipv4.tcp_max_tw_buckets=6000
net.netfilter.nf_conntrack_max=2097152
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF

cat >> containerd.service << EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
[Install]
WantedBy=multi-user.target
EOF

cat >> docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
EOF

cat >> docker.socket << EOF
[Unit]
Description=Docker Socket for the API
PartOf=docker.service
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
EOF


#!/bin/bash
DIR=`pwd`
PACKAGE_NAME="docker-19.03.15.tgz"
DOCKER_FILE=${DIR}/${PACKAGE_NAME}
install_docker(){
  grep "Ubuntu" /etc/issue &> /dev/null
  if [ $? -eq 0 ];then
    /bin/echo  "當(dāng)前系統(tǒng)是`cat /etc/issue`,即將開(kāi)始系統(tǒng)初始化、配置docker-compose與安裝docker" && sleep 1
    \cp ${DIR}/limits.conf /etc/security/limits.conf
    \cp ${DIR}/sysctl.conf /etc/sysctl.conf
    
    /bin/tar xvf ${DOCKER_FILE}
    \cp docker/*  /usr/bin 
    \cp containerd.service /lib/systemd/system/containerd.service
    \cp docker.service  /lib/systemd/system/docker.service
    \cp docker.socket /lib/systemd/system/docker.socket
    \cp ${DIR}/docker-compose-Linux-x86_64_1.24.1 /usr/bin/docker-compose
    ulimit -n 1000000 
    /bin/su -c - [運(yùn)維用戶] "ulimit -n 1000000"
    /bin/echo "docker 安裝完成!" && sleep 1 
    systemctl  enable containerd.service && systemctl  restart containerd.service
    systemctl  enable docker.service && systemctl  restart docker.service
    systemctl  enable docker.socket && systemctl  restart docker.socket 
  fi
};install_docker

1.3 docker鏡像加速

國(guó)內(nèi)下載國(guó)外的鏡像有時(shí)候會(huì)很慢，因此可以更改 docker 配置文件添加一個(gè)加速器，可以通過(guò)加速器達(dá)到加速下載鏡像的目的。

#cat >> /etc/docker/daemon.json << EOF
{
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com",
    "https://mirror.baidubce.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}
EOF

root@harbor2:~# systemctl restart docker
root@harbor2:~# docker info

1.4 docker基礎(chǔ)命令

[root@linux-docker ~]# docker ps
[root@linux-docker ~]# docker ps -a

[root@docker-server1 ~]# docker rm 11445b3a84d3
[root@docker-server1 ~]# docker rm -f 11445b3a84d3

[root@docker-server1 ~]# docker run -P docker.io/nginx
[root@docker-server1 ~]# docker run -p 81:80 --name nginx-test-port1 nginx
[root@docker-server1 ~]# docker run -p 192.168.10.205:83:80/udp --name nginx-test-port4

[root@docker-server1 ~]# docker logs -f nginx-test-port3 #持續(xù)查看

[root@docker-server1 ~]# docker port nginx-test-port5

[root@docker-server1 ~]# docker run -t -i --name test-centos2 docker.io/centos /bin/bash

[root@linux-docker opt]# docker run -it --rm --name nginx-delete-test docker.io/nginx

[root@docker-server1 ~]# docker run -d centos /usr/bin/tail -f '/etc/hosts'

[root@docker-server1 ~]# docker stop f821d0cd5a99 
[root@docker-server1 ~]# docker start f821d0cd5a99

[root@docker-server1 ~]# docker exec -it f821d0cd5a99 bash

[root@docker-server1 ~]# docker ps -a |awk '{print $1}' |xargs docker rm -f
[root@docker-server1 ~]# docker ps -a -q |xargs docker kill
[root@docker-server1 ~]# docker rm -f `docker ps -aq -f status=exited`

2 docker鏡像管理

2.1 鏡像管理命令

// 查看本地的所有鏡像
docker images
// 刪除虛懸鏡像
docker image prune
// 刪除指定鏡像
docker rmi 鏡像名或ID

// 給鏡像打上TAG
docker tag 鏡像ID xxx:xxx

// 將鏡像保存到文件，將文件發(fā)給其他人，其他人再載入即可
docker save 鏡像ID > xxx.tar
// 將文件載入
docker load < xxx.tar

2.2 鏡像構(gòu)建

2.2.1 手動(dòng)構(gòu)建

root@harbor2:~# docker run --name centos-yum-nginx --rm -it centos:7 bash
#在容器里邊安裝以下測(cè)試
    1  yum install -y wget bash-com* epel-rel*
    2  yum install -y nginx
    3  vim /etc/nginx/nginx.conf
    4  yum install -y vim
    5  touch 123.txt
root@harbor2:~# docker commit -a "xxx@qq.com" -m "nginx yum v1" --change="EXPOSE 80 443" centos-yum-nginx centos-nginx:v1
sha256:020e268ddccd5707e82e7cc593be4e78951503351bbde91e2d72e0ffddad61a1  

root@harbor2:~# docker run -it --rm centos-nginx:v1 bash
[root@020e268ddccd /]# ls
123.trxt           bin  etc   lib    media  opt   root  sbin  sys  usr
anaconda-post.log  dev  home  lib64  mnt    proc  run   srv   tmp  var
[root@020e268ddccd /]# exit

2.2.2 Dockerfile構(gòu)建

腳本中的常用語(yǔ)法如下：

FROM 鏡像基于另一個(gè)鏡像，就是在另一個(gè)鏡像的基礎(chǔ)上再執(zhí)行一些腳本構(gòu)建出新的鏡像
MAINTAINER xx.xxx xxx@qq.com 鏡像維護(hù)者的信息 
RUN 執(zhí)行命令
WORKDIR 指定工作目錄，相當(dāng)于cd，一般在Dockerfile結(jié)尾會(huì)將工作目錄切到常用的目錄下，這樣在docker exec進(jìn)入容器時(shí)就會(huì)默認(rèn)進(jìn)入到此目錄下，省去用戶再cd目錄的操作
COPY 復(fù)制當(dāng)前目錄的文件到鏡像中，也可以從另一個(gè)鏡像復(fù)制文件
ADD 添加當(dāng)前目錄的文件到鏡像中，如果要添加的文件是*.tar.gz|tgz格式，則會(huì)自動(dòng)解壓到鏡像的指定目錄下
ENV 設(shè)置鏡像中的環(huán)境變量
CMD 指定鏡像在啟動(dòng)容器時(shí)執(zhí)行的命令，在Dockerfile中CMD只能出現(xiàn)一次
ENTRYPOINT 指定鏡像在啟動(dòng)容器時(shí)執(zhí)行的命令或腳本，由于CMD只能出現(xiàn)一次，如果要在容器啟動(dòng)時(shí)執(zhí)行多條命令就可以用entrypoint代替，可以在鏡像中添加一個(gè).sh文件，在.sh文件中寫多條命令，要注意.sh文件需要有可執(zhí)行權(quán)限，一般會(huì)RUN chmod u+x xxx.sh添加可執(zhí)行權(quán)限。這里給出一條命令的寫法["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/webapps/api.jar", "--spring.profiles.active=test"]
EXPOSE 說(shuō)明暴露的端口，寫不寫都可以，但是寫了之后，在docker inspect查看鏡像時(shí)可以查看到鏡像會(huì)暴露哪些端口
VOLUME 該指令可以指定一個(gè)或多個(gè)目錄作為容器的數(shù)據(jù)卷。容器運(yùn)行時(shí)應(yīng)該盡量保持容器存儲(chǔ)層不發(fā)生寫操作，對(duì)于數(shù)據(jù)庫(kù)類需要保存動(dòng)態(tài)數(shù)據(jù)的應(yīng)用，其數(shù)據(jù)庫(kù)文件應(yīng)該保存于卷(volume)中。為了防止運(yùn)行時(shí)用戶忘記將動(dòng)態(tài)文件所保存目錄掛載為卷，在 Dockerfile 中，我們可以事先指定某些目錄掛載為匿名卷，這樣在運(yùn)行時(shí)如果用戶不指定掛載，其應(yīng)用也可以正常運(yùn)行，不會(huì)向容器存儲(chǔ)層寫入大量數(shù)據(jù)
例如：將 /data 目錄作為容器數(shù)據(jù)卷目錄 /data 目錄就會(huì)在運(yùn)行時(shí)自動(dòng)掛載為匿名卷，任何向 /data 中寫入的信息都不會(huì)記錄進(jìn)容器存儲(chǔ)層，從而保證了容器存儲(chǔ)層的無(wú)狀態(tài)化。當(dāng)然，運(yùn)行時(shí)可以覆蓋這個(gè)掛載設(shè)置。比如
docker run -d -v mydata:/data xxxx 命令中，就使用了 mydata 這個(gè)命名卷掛載到了 /data 這個(gè)位置，替代了 Dockerfile 中定義的匿名卷的掛載配置

springboot 鏡像構(gòu)建

FROM registry.cn-hangzhou.aliyuncs.com/haozheyu/jdk:ora8u201-alpine3.9-glibc2.29


ADD target/x-admin-1.0-SNAPSHOT.jar /x-admin-1.0-SNAPSHOT.jar
EXPOSE 8080

# 指定docker容器啟動(dòng)時(shí)運(yùn)行jar包
ENTRYPOINT ["java", "-jar","/x-admin-1.0-SNAPSHOT.jar"]
# 指定維護(hù)者的名字
MAINTAINER xxx@qq.com

構(gòu)建nginx鏡像

Nginx 鏡像的 DockerFile

FROM centos:7

MAINTAINER xxx<xxx@qq.com>

# 安裝軟件
RUN yum -y update && yum -y install gcc gdb strace gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel ncurses ncurses-devel curl curl-devel e2fsprogs patch e2fsprogs-devel krb5-devel libidn libidn-devel openldap-devel nss_ldap openldap-clients openldap-servers libevent-devel libevent uuid-devel uuid openssl openssl-devel pcre pcre-devel

# 創(chuàng)建用戶
RUN groupadd www
RUN useradd -g www www -s /bin/false

# 定義Nginx版本號(hào)
ENV VERSION 1.14.2

# 下載并解壓文件
RUN mkdir -p /usr/local/src/
ADD http://nginx.org/download/nginx-$VERSION.tar.gz /usr/local/src
RUN tar -xvf /usr/local/src/nginx-$VERSION.tar.gz -C /usr/local/src/

# 創(chuàng)建安裝目錄
ENV NGINX_HOME /usr/local/nginx
RUN mkdir -p $NGINX_HOME
RUN chown -R www:www $NGINX_HOME

# 進(jìn)入解壓目錄
WORKDIR /usr/local/src/nginx-$VERSION

# 編譯安裝
RUN ./configure \
	--user=www \
	--group=www \
	--prefix=$NGINX_HOME \
	--with-http_ssl_module \
	--with-http_realip_module \
	--with-http_gzip_static_module \
	--with-http_stub_status_module
RUN make
RUN make install

# 備份Nginx的配置文件
RUN mv $NGINX_HOME/conf/nginx.conf $NGINX_HOME/conf/nginx.conf.default

# 設(shè)置環(huán)境變量
ENV PATH $PATH:$NGINX_HOME/sbin

# 創(chuàng)建WebApp目錄
ENV WEB_APP /usr/share/nginx/html
RUN mkdir -p $WEB_APP

# 設(shè)置默認(rèn)工作目錄
WORKDIR $WEB_APP

# 暴露端口
EXPOSE 80
EXPOSE 443

# 清理壓縮包與解壓文件
RUN rm -rf /usr/local/src/nginx*

CMD $NGINX_HOME/sbin/nginx -g 'daemon off;' -c $NGINX_HOME/conf/nginx.conf

構(gòu)建 Nginx 鏡像

# 構(gòu)建Nginx鏡像
# docker build -f docker-file -t centos-nginx:1.14.2 .

Docker-Compose 管理 Nginx 鏡像

version: "3.5"

services:
  nginx:
    image: centos-nginx:1.14.2
    container_name: nginx-1.14.2
    privileged: false
    ports:
      - 80:80
      - 443:443
    volumes:
       - '/container/nginx/wwwroot:/usr/share/nginx/html'
       - '/container/nginx/logs:/usr/local/nginx/logs'
       - '/container/nginx/nginx.conf:/usr/local/nginx/conf/nginx.conf'

# 上面的配置是docker-compose.yml文件的內(nèi)容，數(shù)據(jù)卷部分可以根據(jù)自己的實(shí)際情況進(jìn)行修改
# 注意： 在/container/nginx/nginx.conf配置文件中，需要手動(dòng)修改root的路徑為/usr/share/nginx/html

創(chuàng)建并啟動(dòng) Nginx 容器

# 創(chuàng)建并啟動(dòng)容器
# docker-compose up -d

# 查看容器的運(yùn)行狀態(tài)
# docker-compose ps

手動(dòng)構(gòu)建tengine

FROM centos:7

MAINTAINER xxx<xxx@qq.com>

# 安裝軟件
RUN yum -y update && yum -y install vim tree htop tmux net-tools telnet wget curl supervistor autoconf git gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel

# 創(chuàng)建用戶
RUN groupadd tengine
RUN useradd -g tengine tengine

# 定義Tengine版本號(hào)
ENV VERSION 2.2.3

# 下載并解壓文件
RUN mkdir -p /usr/local/src/
ADD http://tengine.taobao.org/download/tengine-$VERSION.tar.gz /usr/local/src
RUN tar -xvf /usr/local/src/tengine-$VERSION.tar.gz -C /usr/local/src/

# 創(chuàng)建安裝目錄
ENV TENGINE_HOME /usr/local/tengine
RUN mkdir -p $TENGINE_HOME

# 進(jìn)入解壓目錄
WORKDIR /usr/local/src/tengine-$VERSION

# 編譯安裝
RUN ./configure \
	--user=tengine \
	--group=tengine \
	--prefix=$TENGINE_HOME \
	--with-http_ssl_module \
	--with-http_realip_module \
	--with-http_concat_module  \
	--with-http_gzip_static_module \
	--with-http_stub_status_module \
	--with-http_upstream_consistent_hash_module
RUN make
RUN make install

# 備份Tengine的配置文件
RUN mv $TENGINE_HOME/conf/nginx.conf $TENGINE_HOME/conf/nginx.conf.default

# 設(shè)置環(huán)境變量
ENV PATH $PATH:$TENGINE_HOME/sbin

# 創(chuàng)建WebApp目錄
ENV WEB_APP /usr/share/tengine/html
RUN mkdir -p $WEB_APP

# 設(shè)置默認(rèn)工作目錄
WORKDIR $WEB_APP

# 暴露端口
EXPOSE 80
EXPOSE 443

# 清理壓縮包與解壓文件
RUN rm -rf /usr/local/src/tengine*

CMD $TENGINE_HOME/sbin/nginx -g 'daemon off;' -c $TENGINE_HOME/conf/nginx.conf

構(gòu)建 Tengine 鏡像

# 構(gòu)建Tengine鏡像
# docker build -f docker-file -t centos-tengine:2.2.3 .

Docker-Compose 管理 Tengine 鏡像

version: "3.5"

services:
  tengine:
    image: centos-tengine:2.2.3
    container_name: tengine:2.2.3
    restart: always
    privileged: false
    ports:
      - 80:80
      - 443:443
    volumes:
       - '/container/tengine/wwwroot/:/usr/share/tengine/html'
       - '/container/tengine/logs:/usr/local/tengine/logs'
       - '/container/tengine/nginx.conf:/usr/local/tengine/conf/nginx.conf'

# 上面的配置是docker-compose.yml文件的內(nèi)容，數(shù)據(jù)卷部分可以根據(jù)自己的實(shí)際情況進(jìn)行修改
# 注意： 在/container/tengine/nginx.conf配置文件中需要手動(dòng)修改root的路徑為/usr/share/tengine/html

創(chuàng)建并啟動(dòng) Tengine 容器

# 創(chuàng)建并啟動(dòng)容器
# docker-compose up -d

# 查看容器的運(yùn)行狀態(tài)
# docker-compose ps

構(gòu)建tomcat 容器

FROM registry.cn-hangzhou.aliyuncs.com/haozheyu/jdk:ora8u201-alpine3.9-glibc2.29
#env 
ENV TZ "Asia/Shanghai" 
ENV LANG en_US.UTF-8 
ENV TERM xterm 
ENV TOMCAT_MAJOR_VERSION 8 
ENV TOMCAT_MINOR_VERSION 8.5.45 
ENV CATALINA_HOME /apps/tomcat 
ENV APP_DIR ${CATALINA_HOME}/webapps 

#tomcat 
RUN mkdir /apps 
ADD apache-tomcat-8.5.45.tar.gz /apps 
RUN ln -sv /apps/apache-tomcat-8.5.45 /apps/tomcat

docker build -t tomcat-bash:v8.5.45 .

3 docker 數(shù)據(jù)管理

Docker 的鏡像是分層設(shè)計(jì)的，鏡像層是只讀的，通過(guò)鏡像啟動(dòng)的容器添加了一層可讀寫的文件系統(tǒng)，用戶寫入的數(shù)據(jù)都保存在這一層當(dāng)中。 如果要將寫入到容器的數(shù)據(jù)永久保存，則需要將容器中的數(shù)據(jù)保存到宿主機(jī)的指定目錄，目前 Docker 的數(shù)據(jù)類型分為兩種:

一是數(shù)據(jù)卷(data volume)，數(shù)據(jù)卷類似于掛載的一塊磁盤，數(shù)據(jù)容器是將數(shù)據(jù)保存在一個(gè)容器上.

二是數(shù)據(jù)卷容器(Data volume container), 數(shù)據(jù)卷容器是將宿主機(jī)的目錄掛載至一個(gè)專門的數(shù)據(jù)卷容器，然后讓其他容器通過(guò)數(shù)據(jù)卷容器讀寫宿主機(jī)的數(shù)據(jù)。

# docker inspect 89e6038bf99f #查看指定 PID 的容器信息
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/119f71309ce1be75aab7760dd04bcb7c02ad6b41725c10d3bb6ed590a945359f-init/diff:/var/lib/docker/overlay2/eb1a488d3821fa84d0fbf871891404c5c6cd68046775225a9f652b1591fd9a2c/diff:/var/lib/docker/overlay2/f8bacdf8595bab5330b8a07d83d823722e817f6219808088ce5ab873371e403d/diff",
                "MergedDir": "/var/lib/docker/overlay2/119f71309ce1be75aab7760dd04bcb7c02ad6b41725c10d3bb6ed590a945359f/merged",
                "UpperDir": "/var/lib/docker/overlay2/119f71309ce1be75aab7760dd04bcb7c02ad6b41725c10d3bb6ed590a945359f/diff",
                "WorkDir": "/var/lib/docker/overlay2/119f71309ce1be75aab7760dd04bcb7c02ad6b41725c10d3bb6ed590a945359f/work"
            },
            "Name": "overlay2"
        },
        
Lower Dir：image 鏡像層(鏡像本身，只讀) 
Upper Dir：容器的上層(讀寫) 
Merged Dir：容器的文件系統(tǒng)，使用 Union FS（聯(lián)合文件系統(tǒng)）將 lowerdir 和 upper Dir：合并給容器使用。 
Work Dir：容器在 宿主機(jī)的工作目錄

數(shù)據(jù)卷實(shí)際上就是宿主機(jī)上的目錄或者是文件，可以被直接 mount 到容器當(dāng)中使用。

3.1 啟動(dòng)測(cè)試容器驗(yàn)證

啟動(dòng)倆個(gè)終端

docker run -it --rm --name test2 -v /tmp:/tmp 53e7a2d8aadd bash
docker run -it --rm --name test1 -v /tmp:/tmp 53e7a2d8aadd bash
[root@2efd6b2984d1 /]# cd /tmp/
[root@2efd6b2984d1 tmp]# touch mytest2
#在另一個(gè)容器
[root@81d6a015cd7c /]# cd /tmp/
#宿主機(jī)的tmp同樣也有
[root@81d6a015cd7c tmp]# ls
mytest2
root@harbor2:/tmp# ls
mytest2

3.2 數(shù)據(jù)卷的特點(diǎn)

1、數(shù)據(jù)卷是宿主機(jī)的目錄或者文件，并且可以在多個(gè)容器之間共同使用。
2、在宿主機(jī)對(duì)數(shù)據(jù)卷更改數(shù)據(jù)后會(huì)在所有容器里面會(huì)立即更新。
3、數(shù)據(jù)卷的數(shù)據(jù)可以持久保存，即使刪除使用使用該容器卷的容器也不影響。
4、在容器里面的寫入數(shù)據(jù)不會(huì)影響到鏡像本身。

3.3 數(shù)據(jù)卷的使用場(chǎng)景

1、日志輸出 
2、靜態(tài) web 頁(yè)面 
3、應(yīng)用配置文件 
4、多容器間目錄或文件共享

3.4 鏡像管理

在容器管理這一節(jié)主要想說(shuō)一下容器占用磁盤空間的問(wèn)題，容器運(yùn)行久了，占用的空間會(huì)越來(lái)越大，當(dāng)一臺(tái)服務(wù)器上運(yùn)行多個(gè)容器時(shí)，可能硬盤突然滿了，此時(shí)就需要能及時(shí)找到哪個(gè)容器占用了過(guò)多的空間，以及如何盡快清理出空間

通過(guò)docker system df命令查看docker占用的磁盤空間大小

TYPE這一列Images是鏡像的大小，Containers是容器占用的大小，Local Volumes是掛載的數(shù)據(jù)卷大小

• 如果鏡像很大，可以docker images查看是否有哪些鏡像不需要，可以刪除
• 如果數(shù)據(jù)卷很大，可能是掛載的目錄被寫入很多文件，如容器中運(yùn)行軟件產(chǎn)生的日志文件或產(chǎn)生的數(shù)據(jù)文件

通過(guò)docker system df -v查看更詳細(xì)的數(shù)據(jù)，包括每個(gè)鏡像占用的空間，每個(gè)容器占用的空間等

通過(guò)docker ps --size查看運(yùn)行中的容器占用的空間

如果docker根目錄所在的硬盤確實(shí)騰不出空間了，可以掛一個(gè)新的硬盤，service docker stop關(guān)閉docker服務(wù)，將當(dāng)前docker根目錄下的文件移動(dòng)到新的硬盤（默認(rèn)docker的根目錄是/var/lib/docker），修改/etc/docker/daemon.json添加或修改data-root到新的硬盤上，再service docker start啟動(dòng)docker服務(wù)，這樣就完成了docker目錄的遷移

3.5 鏡像縮容

構(gòu)建的鏡像當(dāng)然是越小越好，使鏡像中包含的程序剛剛好，這樣構(gòu)建出來(lái)的鏡像才能最小化

docker鏡像是由構(gòu)建的步驟一層一層疊加出來(lái)的，這是linux中自帶的overlay文件系統(tǒng)，如果在第1步添加了一個(gè)文件到鏡像中，在第2步再刪除該文件，這個(gè)文件的大小仍然會(huì)算在鏡像大小中，如果將第1步和第2步合并成一步，那么鏡像大小就不再包含該文件大小了，如下：

RUN 添加文件到鏡像中 \
  && 刪除文件

上面的\是轉(zhuǎn)義換行符的，&&是連接兩個(gè)命令的符號(hào)

docker提供了一個(gè)命令可以查看鏡像的構(gòu)建歷史步驟

docker history 鏡像名或ID

root@harbor2:/tmp# docker history c0fa8ada5a84
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
c0fa8ada5a84        10 months ago       /bin/sh -c #(nop)  CMD ["/usr/local/openrest…   0B                  
62bbb0b2a328        10 months ago       /bin/sh -c #(nop)  EXPOSE 80                    0B                  
e228ad0a338b        10 months ago       /bin/sh -c #(nop) WORKDIR /                     0B                  
b109c44fb8bb        10 months ago       /bin/sh -c #(nop) ADD file:c26c0723c8896ccb9…   857B                
4ed6730165c6        10 months ago       /bin/sh -c make && make install                 95.1MB              
de61b50377e0        10 months ago       /bin/sh -c ./configure --add-module=/usr/loc…   65.2MB              
ddbc24e63d21        10 months ago       /bin/sh -c rm -rf ./Makefile                    0B                  
49631835d627        10 months ago       /bin/sh -c #(nop) WORKDIR /usr/local/src/ope…   0B                  
6e16d85978fd        10 months ago       /bin/sh -c unzip nginx-module-vts.zip           2.58MB              
f393685b4b14        10 months ago       /bin/sh -c #(nop) WORKDIR /usr/local/src        0B                  
f6d9e27571d4        10 months ago       /bin/sh -c #(nop) ADD file:9bb161e63fe3864a4…   1.49MB              
1548553d9fc8        10 months ago       /bin/sh -c #(nop) ADD file:3d10850fb8b3e933a…   27.1MB              
e8b57fcbd31d        10 months ago       /bin/sh -c apt-get install unzip make gcc li…   210MB               
bc016c01c294        10 months ago       /bin/sh -c apt-get update                       26.2MB              
d820629463da        10 months ago       /bin/sh -c echo 'deb http://mirrors.aliyun.c…   351B                
5991db817e28        10 months ago       /bin/sh -c echo 'deb http://mirrors.aliyun.c…   260B                
1f911ec5d70c        10 months ago       /bin/sh -c echo 'deb http://mirrors.aliyun.c…   171B                
65e875ff00b0        10 months ago       /bin/sh -c echo 'deb http://mirrors.aliyun.c…   81B                 
90088fba500c        10 months ago       /bin/sh -c #(nop) WORKDIR /etc/apt              0B                  
fab5e942c505        10 months ago       /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
<missing>           10 months ago       /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B                  
<missing>           10 months ago       /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B                
<missing>           10 months ago       /bin/sh -c rm -rf /var/lib/apt/lists/*          0B                  
<missing>           10 months ago       /bin/sh -c #(nop) ADD file:513ae777bc4042f84…   126MB

通過(guò)這個(gè)命令可以查看到指定鏡像在構(gòu)建時(shí)每一步操作所占用的大小

3.6 多階段構(gòu)建

docker也提供了多階段構(gòu)建的能力，以構(gòu)建Java鏡像為例，我們需要maven來(lái)將源碼編譯成.jar，在容器運(yùn)行時(shí)只需要jre，這樣maven只用于構(gòu)建時(shí)，如果能將maven從鏡像中排除掉，最終構(gòu)建出的java鏡像就會(huì)小很多

# 先基于maven鏡像來(lái)編譯出.jar
FROM maven:3-jdk-8 AS builder
COPY ./src/ /usr/src/java/
WORKDIR /usr/src/java/
RUN mvn clean package

# 再基于openjdk鏡像來(lái)運(yùn)行java，此時(shí)就需要從上面鏡像中復(fù)制編譯好的.jar文件
FROM openjdk:8u102-jre
COPY --from=builder /usr/src/java/target/xxx.jar /usr/local/jar/xxx.jar
CMD ["java", "-jar", "/usr/local/jar/xxx.jar"]

4 Harbor 倉(cāng)庫(kù)使用

Harbor是一個(gè)用于存儲(chǔ)和分發(fā)Docker鏡像的企業(yè)級(jí)Registry服務(wù)器，由vmware開(kāi)源，其通過(guò)添加一些企業(yè)必需的功能特性，例如安全、標(biāo)識(shí)和管理等，擴(kuò)展了開(kāi)源 Docker Distribution。作為一個(gè)企業(yè)級(jí)私有 Registry 服務(wù)器，Harbor 提供了更好的性能和安全。提升用戶使用 Registry 構(gòu)建和運(yùn)行環(huán)境傳輸鏡像的效率。Harbor 支持安裝在多個(gè) Registry 節(jié)點(diǎn)的鏡像資源復(fù)制，鏡像全部保存在私有 Registry 中，確保數(shù)據(jù)和知識(shí)產(chǎn)權(quán)在公司內(nèi)部網(wǎng)絡(luò)中管控，另外，Harbor 也提供了高級(jí)的安全特性，諸如用戶管理，訪問(wèn)控制和活動(dòng)審計(jì)等。

vmware 官方開(kāi)源服務(wù)列表地址：https://vmware.github.io/harbor/cn/，

harbor 官方 github 地址：https://github.com/vmware/harbor

harbor 官方網(wǎng)址：https://goharbor.io/

下載地址：https://github.com/vmware/harbor/releases

安裝文檔：https://goharbor.io/docs/2.3.0/install-config/

4.1 harbor部署

4.1.1 機(jī)器環(huán)境

節(jié)點(diǎn)hostname	host	IP
harbor	reg.local.com	192.168.43.131

4.1.2 hostname

[root@base1 ~]# hostnamectl set-hostname harbor --static

4.1.3 網(wǎng)絡(luò)設(shè)置

[root@base1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
BOOTPROTO="static" #dhcp改為static 
ONBOOT="yes" #開(kāi)機(jī)啟用本配置
IPADDR=192.168.43.131 #靜態(tài)IP
GATEWAY=192.168.43.1 #默認(rèn)網(wǎng)關(guān)
NETMASK=255.255.255.0 #子網(wǎng)掩碼
DNS1=114.114.114.114 #DNS 配置
DNS2=8.8.8.8 #DNS 配置

$# reboot

4.1.4 查看主機(jī)名

hostname

4.1.5 ip:hostname到每一臺(tái)機(jī)器節(jié)點(diǎn)

echo "192.168.43.131 reg.local.com" >> /etc/hosts

4.1.6 安裝依賴環(huán)境，注意：每一臺(tái)機(jī)器都需要安裝此依賴環(huán)境

yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstatlibseccomp wget vim net-tools git iproute lrzsz bash-completion tree bridge-utils unzip bind-utils gcc

4.1.7 docker部署

4.1.7.1 安裝docker

yum install -y yum-utils device-mapper-persistent-data lvm2

#緊接著配置一個(gè)穩(wěn)定的倉(cāng)庫(kù)、倉(cāng)庫(kù)配置會(huì)保存到/etc/yum.repos.d/docker-ce.repo文件中
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
#更新Yum安裝的相關(guān)Docker軟件包&安裝Docker CE
yum update -y && yum install docker-ce

4.1.7.2 設(shè)置docker daemon文件

#創(chuàng)建/etc/docker目錄
mkdir /etc/docker
#更新daemon.json文件
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts":["native.cgroupdriver=systemd"],
"log-driver":"json-file",
"log-opts":{"max-size":"100m"}
}
EOF
#注意：一定注意編碼問(wèn)題，出現(xiàn)錯(cuò)誤---查看命令：journalctl -amu docker 即可發(fā)現(xiàn)錯(cuò)誤
#創(chuàng)建，存儲(chǔ)docker配置文件
mkdir -p /etc/systemd/system/docker.service.d

4.1.7.3 重啟docker服務(wù)

systemctl daemon-reload && systemctl restart docker && systemctl enable docker

4.1.8 安裝compose

打開(kāi)github.com官網(wǎng)，在登錄頁(yè)面的右上角搜索compose找到docker/compose再找releases，（網(wǎng)址：github.com/docker/comp…

復(fù)制自己所需版本下提供的兩條命令，在第一臺(tái)Docker服務(wù)器上依次進(jìn)行操作：

#在線下載docker-compose ,harbor需要借助docker-compose安裝
#復(fù)制官網(wǎng)上的上述命令
curl -L https://github.com/docker/compose/releases/download/1.27.4/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

#賦予該命令執(zhí)行權(quán)限
chmod u+x /usr/local/bin/docker-compose  
#查看其版本信息
docker-compose -version   
docker-compose version 1.24.1, build 4667896b

4.1.9 Harbor安裝

4.1.9.1 下載Harbor并配置

#下載harbor
wget https://github.com/goharbor/harbor/releases/download/v2.1.2/harbor-offline-installer-v2.1.2.tgz
#將下載的安裝包解壓到指定目錄
tar zxf harbor-offline-installer-v2.1.2.tgz -C /usr/local
#切換至解壓后的目錄中
cd /usr/local/harbor/
#編輯這個(gè)配置文件
mv harbor.yml.tmpl harbor.yml

vim harbor.yml

修改harbor.yml配置文件

注意點(diǎn)#TODO

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: reg.local.com|192.168.43.131

# http related config
# http: #TODO
  # port for http, default is 80. If https enabled, this port will redirect to https port
  # port: 80#TODO

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/cert/reg.local.com.crt#TODO
  private_key: /data/cert/reg.local.com.key#TODO

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 50
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
  max_open_conns: 1000

# The default data volume
data_volume: /data

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

# Clair configuration
clair:
  # The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.
  updaters_interval: 12

# Trivy configuration
#
# Trivy DB contains vulnerability information from NVD, Red Hat, and many other upstream vulnerability databases.
# It is downloaded by Trivy from the GitHub release page https://github.com/aquasecurity/trivy-db/releases and cached
# in the local file system. In addition, the database contains the update timestamp so Trivy can detect whether it
# should download a newer version from the Internet or use the cached one. Currently, the database is updated every
# 12 hours and published as a new release to GitHub.
trivy:
  # ignoreUnfixed The flag to display only fixed vulnerabilities
  ignore_unfixed: false
  # skipUpdate The flag to enable or disable Trivy DB downloads from GitHub
  #
  # You might want to enable this flag in test or CI/CD environments to avoid GitHub rate limiting issues.
  # If the flag is enabled you have to download the `trivy-offline.tar.gz` archive manually, extract `trivy.db` and
  # `metadata.json` files and mount them in the `/home/scanner/.cache/trivy/db` path.
  skip_update: false
  #
  # insecure The flag to skip verifying registry certificate
  insecure: false
  # github_token The GitHub access token to download Trivy DB
  #
  # Anonymous downloads from GitHub are subject to the limit of 60 requests per hour. Normally such rate limit is enough
  # for production operations. If, for any reason, it's not enough, you could increase the rate limit to 5000
  # requests per hour by specifying the GitHub access token. For more details on GitHub rate limiting please consult
  # https://developer.github.com/v3/#rate-limiting
  #
  # You can create a GitHub token by following the instructions in
  # https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
  #
  # github_token: xxx

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 10

chart:
  # Change the value of absolute_url to enabled can enable absolute url in chart
  absolute_url: disabled

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.0.0

# Uncomment external_database if using external database.
# external_database:
#   harbor:
#     host: harbor_db_host
#     port: harbor_db_port
#     db_name: harbor_db_name
#     username: harbor_db_username
#     password: harbor_db_password
#     ssl_mode: disable
#     max_idle_conns: 2
#     max_open_conns: 0
#   clair:
#     host: clair_db_host
#     port: clair_db_port
#     db_name: clair_db_name
#     username: clair_db_username
#     password: clair_db_password
#     ssl_mode: disable
#   notary_signer:
#     host: notary_signer_db_host
#     port: notary_signer_db_port
#     db_name: notary_signer_db_name
#     username: notary_signer_db_username
#     password: notary_signer_db_password
#     ssl_mode: disable
#   notary_server:
#     host: notary_server_db_host
#     port: notary_server_db_port
#     db_name: notary_server_db_name
#     username: notary_server_db_username
#     password: notary_server_db_password
#     ssl_mode: disable

# Uncomment external_redis if using external Redis server
# external_redis:
#   # support redis, redis+sentinel
#   # host for redis: <host_redis>:<port_redis>
#   # host for redis+sentinel:
#   #  <host_sentinel1>:<port_sentinel1>,<host_sentinel2>:<port_sentinel2>,<host_sentinel3>:<port_sentinel3>
#   host: redis:6379
#   password:
#   # sentinel_master_set must be set to support redis+sentinel
#   #sentinel_master_set:
#   # db_index 0 is for core, it's unchangeable
#   registry_db_index: 1
#   jobservice_db_index: 2
#   chartmuseum_db_index: 3
#   clair_db_index: 4
#   trivy_db_index: 5
#   idle_timeout_seconds: 30

# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
#   ca_file: /path/to/ca

# Global proxy
# Config http proxy for components, e.g. http://my.proxy.com:3128
# Components doesn't need to connect to each others via http proxy.
# Remove component from `components` array if want disable proxy
# for it. If you want use proxy for replication, MUST enable proxy
# for core and jobservice, and set `http_proxy` and `https_proxy`.
# Add domain to the `no_proxy` field, when you want disable proxy
# for some special registry.
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - clair
    - trivy

4.1.9.2 生成證書

一鍵腳本文件create_cert.sh

#!/bin/bash

# 在該目錄下操作生成證書，正好供harbor.yml使用
mkdir -p /data/cert
cd /data/cert

openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=reg.local.com" -key ca.key -out ca.crt
openssl genrsa -out reg.local.com.key 4096
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=reg.local.com" -key reg.local.com.key -out reg.local.com.csr

cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=reg.local.com
DNS.2=harbor
DNS.3=ks-allinone
EOF

openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in reg.local.com.csr -out reg.local.com.crt
    
openssl x509 -inform PEM -in reg.local.com.crt -out reg.local.com.cert

cp reg.local.com.crt /etc/pki/ca-trust/source/anchors/reg.local.com.crt 
update-ca-trust

執(zhí)行腳本，生成證書

chmod 755 create_cert.sh
./create_cert.sh

4.1.9.3 安裝

#執(zhí)行自帶的安裝腳本，安裝完畢，瀏覽器即可訪問(wèn)
./install.sh
...
[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-db     ... done
Creating registry      ... done
Creating registryctl   ... done
Creating redis         ... done
Creating harbor-portal ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
? ----Harbor has been installed and started successfully.----

4.1.9.4 更新daemon.json文件

cat > /etc/docker/daemon.json <<EOF
{
"exec-opts":["native.cgroupdriver=systemd"],
"log-driver":"json-file","log-opts":{"max-size":"100m"},
"registry-mirrors":["https://pee6w651.mirror.aliyuncs.com"],
"insecure-registries": ["https://reg.local.com"]
}
EOF

#確定80端口正在監(jiān)聽(tīng)
netstat -antp | grep 80 

#重啟docker
systemctl daemon-reload && systemctl restart docker

#重啟所有容器
cd /usr/local/harbor
docker-compose stop && docker-compose start
Stopping harbor-jobservice ... done
Stopping nginx             ... done
Stopping harbor-core       ... done
Stopping harbor-portal     ... done
Stopping redis             ... done
Stopping registryctl       ... done
Stopping registry          ... done
Stopping harbor-db         ... done
Stopping harbor-log        ... done
Starting log         ... done
Starting registry    ... done
Starting registryctl ... done
Starting postgresql  ... done
Starting portal      ... done
Starting redis       ... done
Starting core        ... done
Starting jobservice  ... done
Starting proxy       ... done

啟動(dòng)驗(yàn)證Harbor（admin/Harbor12345）

4.2 harbor 上傳和下載鏡像

注意：如果我們配置的是 https 的話，本地 docker 就不需要有任何操作就可以訪問(wèn) harbor 了

4.2.1 node節(jié)點(diǎn)添加私有倉(cāng)庫(kù)

# cat /etc/docker/daemon.json 
{
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com",
    "https://mirror.baidubce.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker",
  "insecure-registries": [
    "xx.xx.xx.xx",
    "xx.xx.xx.xx"
  ]
}

or-------------------------------------------
# cat /etc/systemd/system/multi-user.target.wants/docker.service
Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry=xx.xx.xx.xx

4.2.2 驗(yàn)證是否可以登錄

# docker login 192.168.43.131 --username=admin --password=Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

4.2.3 鏡像打tag推送

# docker tag centos-nginx:v1 192.168.43.131/library/centos-nginx:v1
# docker push 192.168.43.131/library/centos-nginx:v1

4.3 實(shí)現(xiàn)harbor的高可用

Harbor 支持基于策略的 Docker 鏡像復(fù)制功能，這類似于 MySQL 的主從同步， 其可以實(shí)現(xiàn)不同的數(shù)據(jù)中心、不同的運(yùn)行環(huán)境之間同步鏡像

4.3.1 準(zhǔn)備運(yùn)行環(huán)境

192.168.43.66	harbor1	admin	Harbor12345
192.168.43.67	harbor2	admin	Harbor12345

4.3.2 添加倉(cāng)庫(kù)和復(fù)制規(guī)則

#192.168.43.66
#1 系統(tǒng)管理-倉(cāng)庫(kù)管理-新建目標(biāo)
目標(biāo)名稱：67
目標(biāo)url：http://192.168.43.67
訪問(wèn)ID：admin
訪問(wèn)密碼：Harbor12345
#2 系統(tǒng)管理-復(fù)制管理-新建目錄
名稱：push-67
復(fù)制模式：push-based
源資源過(guò)濾器：全部
目標(biāo)倉(cāng)庫(kù)：67-http://192.168.43.67
觸發(fā)模式：事件驅(qū)動(dòng)

#192.168.43.67
#1 系統(tǒng)管理-倉(cāng)庫(kù)管理-新建目標(biāo)
目標(biāo)名稱：66
目標(biāo)url：http://192.168.43.66
訪問(wèn)ID：admin
訪問(wèn)密碼：Harbor12345
#2 系統(tǒng)管理-復(fù)制管理-新建目錄
名稱：push-66
復(fù)制模式：push-based
源資源過(guò)濾器：全部
目標(biāo)倉(cāng)庫(kù)：66-http://192.168.43.66
觸發(fā)模式：事件驅(qū)動(dòng)

4.3.3 驗(yàn)證復(fù)制是否成功

#1 將鏡像push到66的goapp項(xiàng)目下
root@harbor2:~# docker tag registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 192.168.43.66/goapp/centos-stress:11
root@harbor2:~# docker push 192.168.43.66/goapp/centos-stress:11
The push refers to repository [192.168.43.66/goapp/centos-stress]
b2afbca5d0cd: Pushed 
174f56854903: Mounted from goapp/centos 
11: digest: sha256:fac58e4d667483e16f2be89dce8cc8112e5aec8e79ba76d7cd83889c600446a9 size: 741

#2 在67上拉取192.168.43.66/goapp/centos-stress:11
root@harbor:~/harbor# docker pull 192.168.43.67/goapp/centos-stress:11
11: Pulling from goapp/centos-stress
2d473b07cdd5: Pull complete 
97002bbdba0c: Pull complete 
Digest: sha256:fac58e4d667483e16f2be89dce8cc8112e5aec8e79ba76d7cd83889c600446a9
Status: Downloaded newer image for 192.168.43.67/goapp/centos-stress:11

4.3.4 配置harbor的代理

配置使用的VIP：192.168.43.250

4.3.4.1 配置keepalived

#harbor1 harbor2 
apt install -y keepalive haproxy
#harbor1 
cat >> /etc/keepalived/keepalived.conf << EOF
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 1
    priority 100
    advert_int 3
    unicast_src_ip 192.168.43.66
    unicast_peer {
    192.168.43.67
    }
    authentication {
    auth_type PASS
    auth_pass 123abc
    }
    virtual_ipaddress {
    192.168.43.250 dev ens33 label ens33:1
    }
}
EOF

#harbor2
cat >> /etc/keepalived/keepalived.conf << EOF
vrrp_instance VI_1 {
    state SLAVE
    interface ens33
    virtual_router_id 1
    priority 100
    advert_int 3
    unicast_src_ip 192.168.43.67
    unicast_peer {
    192.168.43.66
    }
    authentication {
    auth_type PASS
    auth_pass 123abc
    }
    virtual_ipaddress {
    192.168.43.250 dev ens33 label ens33:1
    }
}
EOF

#harbor1 harbo2 啟動(dòng)服務(wù)
root@harbor:~/harbor# systemctl start keepalived.service
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:2d:30:86 brd ff:ff:ff:ff:ff:ff
    inet 192.168.43.66/24 brd 192.168.43.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet `192.168.43.250/32` scope global ens33:1
       valid_lft forever preferred_lft forever
    inet6 240e:418:400:8f6:20c:29ff:fe2d:3086/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3124sec preferred_lft 3124sec
    inet6 fe80::20c:29ff:fe2d:3086/64 scope link 
       valid_lft forever preferred_lft forever
       
#這里看到VIP已經(jīng)起來(lái)了   

4.3.4.2 配置haproxy

#harbor1 harbor2 追加 /etc/haproxy/haproxy.cfg
listen harbor_80
    bind 192.168.43.250:8180
    mode tcp
    balance source
    server 192.168.43.66 192.168.43.66:80 check inter 2000 fall 3 rise 5
    server 192.168.43.67 192.168.43.67:80 check inter 2000 fall 3 rise 5
#harbor1 harbor2 啟動(dòng)服務(wù)
systemctl start haproxy.service

4.3.4.3 驗(yàn)證配置

root@harbor:~/harbor# cat /etc/docker/daemon.json 
{
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker",
    "insecure-registries": [
    "192.168.43.66",
    "192.168.43.67",
    "http://192.168.43.250:8180"
		      ]
}

root@harbor:~/harbor# systemctl restart docker
root@harbor:~/harbor# docker login 192.168.43.250:8180 --username=admin --password=Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

root@harbor:~/harbor# docker pull 192.168.43.250:8180/goapp/centos-stress:11
11: Pulling from goapp/centos-stress
Digest: sha256:fac58e4d667483e16f2be89dce8cc8112e5aec8e79ba76d7cd83889c600446a9
Status: Downloaded newer image for 192.168.43.250:8180/goapp/centos-stress:11
192.168.43.250:8180/goapp/centos-stress:11
root@harbor:~/harbor# docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
192.168.43.250:8180/goapp/centos-stress   11                  dc9ee26d0c43        8 hours ago         377MB
192.168.43.67/goapp/centos-stress         11                  dc9ee26d0c43        8 hours ago         377MB

4.3.5 harbor的配置文件解析

hostname: 192.168.43.66 #ip或域名

http:
  port: 80
https:
  port: 443
  certificate: /your/certificate/path
  private_key: /your/private/key/path
harbor_admin_password: Harbor12345

database:  #pgsql的鏈接配置  
  password: root123
  max_idle_conns: 100
  max_open_conns: 900
  
data_volume: /data  #數(shù)據(jù)存儲(chǔ)路徑這里可以使用共享存儲(chǔ)，也是高可用的一種實(shí)現(xiàn)

# 12 hours and published as a new release to GitHub.
trivy:
  ignore_unfixed: false
  skip_update: false
  #
  # insecure The flag to skip verifying registry certificate
  insecure: false

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 10

chart:
  # Change the value of absolute_url to enabled can enable absolute url in chart
  absolute_url: disabled

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.3.0

# 使用外部數(shù)據(jù)庫(kù)
# Uncomment external_database if using external database.
# external_database:
#   harbor:
#     host: harbor_db_host
#     port: harbor_db_port
#     db_name: harbor_db_name
#     username: harbor_db_username
#     password: harbor_db_password
#     ssl_mode: disable
#     max_idle_conns: 2
#     max_open_conns: 0
#   notary_signer:
#     host: notary_signer_db_host
#     port: notary_signer_db_port
#     db_name: notary_signer_db_name
#     username: notary_signer_db_username
#     password: notary_signer_db_password
#     ssl_mode: disable
#   notary_server:
#     host: notary_server_db_host
#     port: notary_server_db_port
#     db_name: notary_server_db_name
#     username: notary_server_db_username
#     password: notary_server_db_password
#     ssl_mode: disable

# 使用外部redis
# Uncomment external_redis if using external Redis server
# external_redis:
#   # support redis, redis+sentinel
#   # host for redis: <host_redis>:<port_redis>
#   # host for redis+sentinel:
#   #  <host_sentinel1>:<port_sentinel1>,<host_sentinel2>:<port_sentinel2>,<host_sentinel3>:<port_sentinel3>
#   host: redis:6379
#   password:
#   # sentinel_master_set must be set to support redis+sentinel
#   #sentinel_master_set:
#   # db_index 0 is for core, it's unchangeable
#   registry_db_index: 1
#   jobservice_db_index: 2
#   chartmuseum_db_index: 3
#   trivy_db_index: 5
#   idle_timeout_seconds: 30

# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
#   ca_file: /path/to/ca

# Global proxy
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy

# 狀態(tài)接口
# metric:
#   enabled: false
#   port: 9090
#   path: /metrics