Etcd 是一个可靠的分布式键值存储, 常用于分布式系统关键数据的存储；而 etcdadm 是一个用于操作 etcd 集群的命令行工具，它可以轻松创建集群、向现有集群添加成员、从现有集群中删除成员等操作；其使用方式类似 kubeadm, 即主要操作流程为: 先启动第一个集群节点，后续节点直接 join 即可
建议通过 PC 端，访问 运维技术帮 (https://ywjsbang.com) 以获取更好阅读体验，由于精力有限，该文章的后续更新、完善仅限此处，望理解 !!
测试环境
| 节点主机名 | 节点 IP 地址 | 系统版本 | etcd 版本 | etcdadm 版本 |
|---|---|---|---|---|
| c7 | 192.168.31.37 | CentOS 7.9.2009 ( 5.4.180-1.el7 ) | V3.5.5 | V0.1.5 |
| c8 | 192.168.31.38 | 同上 | 同上 | 同上 |
| c9 | 192.168.31.39 | 同上 | 同上 | 同上 |
安装 etcdadm
1、预编译二进制安装
wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.5/etcdadm-linux-amd64 mv etcdadm-linux-amd64 /usr/local/bin/etcdadm chmod +x /usr/local/bin/etcdadm scp /usr/local/bin/etcdadm 192.168.31.{38,39}:/usr/local/bin/ 2、各节点系统防火墙放行端口 2379，2380
firewall-cmd --add-port=2379/tcp firewall-cmd --add-port=2380/tcp 初始化 etcd 节点
1、初始化第一个 etcd 集群节点
etcdadm init --version "3.5.5" --init-system "systemd" --install-dir "/opt/bin/" --certs-dir "/etc/etcd/pki" --data-dir "/var/lib/etcd" --release-url "https://github.com/etcd-io/etcd/releases/download" # 主要选项解析 --version # 指定部署的 etcd 版本 --init-system # 设置 etcd 进程管理方式，默认 systemd，取值 kubelet 时，则以容器方法运行 etcd 进程 --install-dir # etcd 二进制程序安装目录 2、etcdadm init 初始化过程解析
# 下载解压、安装二进制文件 etcd、etcdctl 2022-10-20 14:26:12.781166 I | [install] Artifact not found in cache. Trying to fetch from upstream: https://github.com/etcd-io/etcd/releases/download INFO[0000] [install] Downloading & installing etcd https://github.com/etcd-io/etcd/releases/download from 3.5.5 to /var/cache/etcdadm/etcd/v3.5.5 INFO[0000] [install] downloading etcd from https://github.com/etcd-io/etcd/releases/download/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz to /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz INFO[0009] [install] extracting etcd archive /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz to /tmp/etcd641204404 INFO[0009] [install] verifying etcd 3.5.5 is installed in /opt/bin/ # 生成一个自签名的 CA 证书及私钥 INFO[0001] [certificates] creating PKI assets INFO[0001] creating a self signed etcd CA certificate and key files [certificates] Generated ca certificate and key. > /etc/etcd/pki/ca.crt > /etc/etcd/pki/ca.key # 生成一个 server 证书及私钥 INFO[0001] creating a new server certificate and key files for etcd [certificates] Generated server certificate and key. [certificates] server serving cert is signed for DNS names [c7] and IPs [192.168.31.37 127.0.0.1] # > /etc/etcd/pki/server.crt # > /etc/etcd/pki/server.key # 生成一个 peer 证书及私钥 INFO[0001] creating a new certificate and key files for etcd peering [certificates] Generated peer certificate and key. [certificates] peer serving cert is signed for DNS names [c7] and IPs [192.168.31.37] # > /etc/etcd/pki/peer.crt # > /etc/etcd/pki/peer.key # 生成一个用于 etcdctl 的 client 证书及私钥 INFO[0001] creating a new client certificate for the etcdctl [certificates] Generated etcdctl-etcd-client certificate and key. # > /etc/etcd/pki/etcdctl-etcd-client.crt # > /etc/etcd/pki/etcdctl-etcd-client.key # 生成一个用于 k8s apiserver 调用 etcd 时的 client 证书及私钥 INFO[0002] creating a new client certificate for the apiserver calling etcd [certificates] Generated apiserver-etcd-client certificate and key. [certificates] valid certificates and keys now exist in "/etc/etcd/pki" # > /etc/etcd/pki/apiserver-etcd-client.crt # > /etc/etcd/pki/apiserver-etcd-client.key # 检查本地 etcd 端点是否健康 INFO[0003] [health] Checking local etcd endpoint health INFO[0003] [health] Local etcd endpoint is healthy # 复制 CA cert/key 到其它 etcd 节点，并在其它 etcd 节点运行 etcdadm join 命令， 将其它 etcd 节点加入集群 INFO[0003] To add another member to the cluster, copy the CA cert/key to its certificate dir and run: INFO[0003] etcdadm join https://192.168.31.37:2379 3、向其它节点分发 CA 根证书及私钥
ssh root@192.168.31.38 "mkdir /etc/etcd/pki/" scp -r /etc/etcd/pki/{ca.crt,ca.key} 192.168.31.38:/etc/etcd/pki/ ssh root@192.168.31.39 "mkdir /etc/etcd/pki/" scp -r /etc/etcd/pki/{ca.crt,ca.key} 192.168.31.39:/etc/etcd/pki/ 添加 etcd 节点
若当前主机无法下载，可提前将 etcd 二进制程序包存放在如下路径: /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz
1、添加节点 192.168.31.38
etcdadm join https://192.168.31.38:2379 --version "3.5.5" --init-system "systemd" --install-dir "/opt/bin/" --certs-dir "/etc/etcd/pki" --data-dir "/var/lib/etcd" --release-url "https://github.com/etcd-io/etcd/releases/download" 2、添加节点 192.168.31.39
etcdadm join https://192.168.31.38:2379 --version "3.5.5" --init-system "systemd" --install-dir "/opt/bin/" --certs-dir "/etc/etcd/pki" --data-dir "/var/lib/etcd" --release-url "https://github.com/etcd-io/etcd/releases/download" Etcd Server
1、用于 Etcd Server 的环境变量配置 /etc/etcd/etcd.env
ETCD_NAME=c7 # Initial cluster configuration ETCD_INITIAL_CLUSTER=c7=https://192.168.31.37:2380 ETCD_INITIAL_CLUSTER_TOKEN=dee8095f ETCD_INITIAL_CLUSTER_STATE=new # Peer configuration ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.31.37:2380 ETCD_LISTEN_PEER_URLS=https://192.168.31.37:2380 ETCD_CLIENT_CERT_AUTH=true ETCD_PEER_CERT_FILE=/etc/etcd/pki/peer.crt ETCD_PEER_KEY_FILE=/etc/etcd/pki/peer.key ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt # Client/server configuration ETCD_ADVERTISE_CLIENT_URLS=https://192.168.31.37:2379 ETCD_LISTEN_CLIENT_URLS=https://192.168.31.37:2379,https://127.0.0.1:2379 ETCD_PEER_CLIENT_CERT_AUTH=true ETCD_CERT_FILE=/etc/etcd/pki/server.crt ETCD_KEY_FILE=/etc/etcd/pki/server.key ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt # Other ETCD_DATA_DIR=/var/lib/etcd ETCD_STRICT_RECONFIG_CHECK=true GOMAXPROCS=8 # Logging configuration # Profiling/metrics 2、Etcd Server 启动脚本
# cat /etc/systemd/system/etcd.service [Unit] Description=etcd Documentation=https://github.com/coreos/etcd Conflicts=etcd-member.service Conflicts=etcd2.service [Service] EnvironmentFile=/etc/etcd/etcd.env ExecStart=/opt/bin/etcd Type=notify TimeoutStartSec=0 Restart=on-failure RestartSec=5s LimitNOFILE=65536 Nice=-10 IOSchedulingClass=best-effort IOSchedulingPriority=2 MemoryLow=200M [Install] WantedBy=multi-user.target etcdctl.sh
1、用于 etcdctl 的环境变量配置 /etc/etcd/etcdctl.env
export ETCDCTL_API=3 export ETCDCTL_CACERT=/etc/etcd/pki/ca.crt export ETCDCTL_CERT=/etc/etcd/pki/etcdctl-etcd-client.crt export ETCDCTL_KEY=/etc/etcd/pki/etcdctl-etcd-client.key export ETCDCTL_DIAL_TIMEOUT=3s 2、脚本 etcdctl.sh 是对 etcdctl 命令的简单包装，其用法与 etcdctl 一致
cat /opt/bin/etcdctl.sh #!/usr/bin/env sh if ! [ -r "/etc/etcd/etcdctl.env" ]; then echo "Unable to read the etcdctl environment file '/etc/etcd/etcdctl.env'. The file must exist, and this wrapper must be run as root." exit 1 fi . "/etc/etcd/etcdctl.env" # 相当于 source 该环境变量配置文件 "/opt/bin/etcdctl" "$@" # $@ 表示脚本 etcdctl.sh 的命令行参数 管理命令
# 查看命令行 init 或 join 的帮助信息 etcdadm init|join --help # 从 etcd 集群移除当前节点 etcdadm reset # 查看集群节点成员 /opt/bin/etcdctl.sh member list # > 19fc11a542653f62, started, c9, https://192.168.31.39:2380, https://192.168.31.39:2379, false # > 9a246c6786d36273, started, c7, https://192.168.31.37:2380, https://192.168.31.37:2379, false # > a509d3d8e8aa4911, started, c8, https://192.168.31.38:2380, https://192.168.31.38:2379, false # 查看当前节点是否正常 /opt/bin/etcdctl.sh endpoint health # 127.0.0.1:2379 is healthy: successfully committed proposal: took = 17.112587ms # 查看当前节点状态 /opt/bin/etcdctl.sh endpoint status # > 127.0.0.1:2379, 9a246c6786d36273, 3.5.5, 20 kB, true, false, 3, 10, 10, 由于笔者时间、视野、认知有限，本文难免出现错误、疏漏等问题，期待各位读者朋友、业界大佬指正交流, 共同进步 !!
