etcd多台部署，启用https以及ca自签名

etcd多台部署，启用https以及ca自签名

前言

环境要求

1、三台安装centos7的服务器 2、具备访问互联网 3、关闭服务器的防火墙以及selinux

服务器拓扑

etcd集群拓扑

服务器部署组件以及IP清单

CFSSL工具的安装

下载CFSSL的可执行二进制文件

wget wget wget +x cfssl*

下载执行过程截图如下

将这三个二进制可执行文件，修改名称复制到/usr/local/bin/下

cp -v cfssl_linux-amd64 /usr/local/bin/cfsslcp -v cfssljson_linux-amd64 /usr/local/bin/cfssljsoncp -v cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfols /usr/local/bin/cfssl*

使用CFSSL创建CA证书以及etcd的TLS认证证书

创建 CA (Certificate Authority)

创建 CA 配置文件（ca-config.json）

[root@server81 etcdSSl]# vim ca-config.json{ "signing": { "default": { "expiry": "876000h" }, "profiles": { "etcd": { "usages": [ "signing", "key encipherment", "default": { "expiry": "876000h" }, "profiles": { "etcd": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "876000h" } } }}

"字段说明""ca-config.json"：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；"signing"：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；"server auth"：表示client可以用该 CA 对server提供的证书进行验证；"client auth"：表示server可以用该CA对client提供的证书进行验证；

创建 CA 证书签名请求（ca-csr.json）

[root@server81 etcdSSl]# vim ca-csr.json{ "CN": "etcd", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "shenzhen", "L": "shenzhen", "O": "etcd", "OU": "System" } ]}

"CN"：Common Name，etcd 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；"O"：Organization，etcd 从证书中提取该字段作为请求用户所属的组 (Group)；这两个参数在后面的kubernetes启用RBAC模式中很重要，因为需要设置kubelet、admin等角色权限，那么在配置证书的时候就必须配置对了，具体后面在部署kubernetes的时候会进行讲解。"在etcd这两个参数没太大的重要意义，跟着配置就好。"

生成 CA 证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca说明：生成 "ca-csr.json ca-key.pem ca.pem" 三个文件

创建etcd的TLS认证证书

创建 etcd证书签名请求（etcd-csr.json）

[root@server81 etcdSSl]# vim etcd-csr.json{ "CN": "etcd", "hosts": [ "127.0.0.1", "172.16.5.81", "172.16.5.86", "172.16.5.87" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "shenzhen", "L": "shenzhen", "O": "etcd", "OU": "System" } ]}

[^_^]: 如果 hosts 字段不为空，则需要指定授权使用该证书的 IP 或域名列表。[>_<]: 该证书被 etcd 集群使用，所以填写三台etcd服务器的IP即可。

生成 etcd证书和私钥

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd[^_^]: 生成 "etcd-csr.json etcd-key.pem etcd.pem" 三个文件。

将TLS 认证文件拷贝至证书目录下

mkdir -p /etc/etcd/etcdSSLcp * /etc/etcd/etcdSSL[^_^]: 存在CA证书的路径"/etc/etcd/etcdSSL"是自定义的，看个人习惯创建即可。

Server81 安装etcd服务

yum install -y etcd

配置 etcd 的 service文件（/usr/lib/systemd/system）

root@server81 ~]# vim /usr/lib/systemd/system/etcd.service [Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.target[Service]Type=notifyWorkingDirectory=/var/lib/etcd/EnvironmentFile=-/etc/etcd/etcd.conf# set GOMAXPROCS to number of processorsExecStart=/usr/bin/etcd \ --name ${ETCD_NAME} \ --cert-file=/etc/etcd/etcdSSL/etcd.pem \ --key-file=/etc/etcd/etcdSSL/etcd-key.pem \ --peer-cert-file=/etc/etcd/etcdSSL/etcd.pem \ --peer-key-file=/etc/etcd/etcdSSL/etcd-key.pem \ --trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \ --peer-trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \ --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},\ --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster infra1=\ --initial-cluster-state new \ --data-dir=${ETCD_DATA_DIR}Restart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.target

参数说明：1、指定 etcd 的工作目录为 /var/lib/etcd，数据目录为 /var/lib/etcd，需在启动服务前创建这两个目录；在配置中的命令是这条：WorkingDirectory=/var/lib/etcd/2、为了保证通信安全，需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书（trusted-ca-file）；在配置中添加etcd证书的命令是以下： --cert-file=/etc/etcd/etcdSSL/etcd.pem \ --key-file=/etc/etcd/etcdSSL/etcd-key.pem \ --peer-cert-file=/etc/etcd/etcdSSL/etcd.pem \ --peer-key-file=/etc/etcd/etcdSSL/etcd-key.pem \ --trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \ --peer-trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \3、配置etcd的endpoint：--initial-cluster infra1=\4、配置etcd的监听服务集群： --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},\ --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \5、配置etcd创建的集群为新集群，则定义集群状态为new --initial-cluster-state 值为 new6、定义etcd节点的名称，该名称等下从配置文件中获取： --name ${ETCD_NAME} \ 其中配置文件：EnvironmentFile=-/etc/etcd/etcd.conf

etcd的配置文件（/etc/etcd/etcd.conf）

[root@server81 ~]# vim /etc/etcd/etcd.conf#[member]ETCD_NAME=infra1ETCD_DATA_DIR="/var/lib/etcd"ETCD_LISTEN_PEER_URLS="etcd 服务

systemctl daemon-reloadsystemctl start etcdsystemctl status etcd

到了这里基本etcd的证书以及Server81的etcd节点服务已经配置好了，下面就是快速部署配置一下Server86/87两台服务器的etcd证书以及服务。

Server86安装etcd服务

yum install -y etcd

配置 etcd 的 service文件（/usr/lib/systemd/system）

[root@server86 ~]# vim /usr/lib/systemd/system/etcd.service[Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.target[Service]Type=notifyWorkingDirectory=/var/lib/etcd/EnvironmentFile=-/etc/etcd/etcd.conf# set GOMAXPROCS to number of processorsExecStart=/usr/bin/etcd \ --name ${ETCD_NAME} \ --cert-file=/etc/etcd/etcdSSL/etcd.pem \ --key-file=/etc/etcd/etcdSSL/etcd-key.pem \ --peer-cert-file=/etc/etcd/etcdSSL/etcd.pem \ --peer-key-file=/etc/etcd/etcdSSL/etcd-key.pem \ --trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \ --peer-trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \ --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},\ --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster infra1=\ --initial-cluster-state new \ --data-dir=${ETCD_DATA_DIR}Restart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.target

etcd的配置文件（/etc/etcd/etcd.conf）

[root@server86 ~]# vim /etc/etcd/etcd.conf#[member]ETCD_NAME=infra2ETCD_DATA_DIR="/var/lib/etcd"ETCD_LISTEN_PEER_URLS="etc]# cd etcd/[root@server81 etcd]# lsetcd.conf etcdSSL[root@server81 etcd]# [root@server81 etcd]# pwd/etc/etcd[root@server81 etcd]# [root@server81 etcd]# scp -r etcdSSL root@server86:/etc/etcdca-config.json 100% 288 82.3KB/s 00:00 ca.csr 100% 997 283.1KB/s 00:00 ca-csr.json 100% 205 60.5KB/s 00:00 ca-key.pem 100% 1675 395.4KB/s 00:00 ca.pem 100% 1350 356.7KB/s 00:00 etcd.csr 100% 1066 50.8KB/s 00:00 etcd-csr.json 100% 296 40.1KB/s 00:00 etcd-key.pem 100% 1675 356.8KB/s 00:00 etcd.pem 100% 1436 341.3KB/s 00:00 [root@server81 etcd]#

启动 etcd 服务

systemctl daemon-reloadsystemctl start etcdsystemctl status etcd

Server87安装etcd服务

yum install -y etcd

配置 etcd 的 service文件（/usr/lib/systemd/system）

[root@server87 ~]# vim /usr/lib/systemd/system/etcd.service[Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.target[Service]Type=notifyWorkingDirectory=/var/lib/etcd/EnvironmentFile=-/etc/etcd/etcd.conf# set GOMAXPROCS to number of processorsExecStart=/usr/bin/etcd \ --name ${ETCD_NAME} \ --cert-file=/etc/etcd/etcdSSL/etcd.pem \ --key-file=/etc/etcd/etcdSSL/etcd-key.pem \ --peer-cert-file=/etc/etcd/etcdSSL/etcd.pem \ --peer-key-file=/etc/etcd/etcdSSL/etcd-key.pem \ --trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \ --peer-trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \ --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},\ --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster infra1=\ --initial-cluster-state new \ --data-dir=${ETCD_DATA_DIR}Restart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.target

etcd的配置文件（/etc/etcd/etcd.conf）

[root@server87 ~]# vim /etc/etcd/etcd.conf#[member]ETCD_NAME=infra3ETCD_DATA_DIR="/var/lib/etcd"ETCD_LISTEN_PEER_URLS="etcd]# scp -r etcdSSL root@server87:/etc/etcdca-config.json 100% 288 109.8KB/s 00:00 ca.csr 100% 997 417.5KB/s 00:00 ca-csr.json 100% 205 79.4KB/s 00:00 ca-key.pem 100% 1675 575.7KB/s 00:00 ca.pem 100% 1350 485.5KB/s 00:00 etcd.csr 100% 1066 289.0KB/s 00:00 etcd-csr.json 100% 296 120.7KB/s 00:00 etcd-key.pem 100% 1675 601.9KB/s 00:00 etcd.pem 100% 1436 104.4KB/s 00:00 [root@server81 etcd]#

启动 etcd 服务

systemctl daemon-reloadsystemctl start etcdsystemctl status etcd

最后回到server81服务器，检查一下etcd集群是否正常

检查刚才启动etcd失败的Server81的节点

[root@server81 etcd]# service etcd statusRedirecting to /bin/systemctl status etcd.service● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2018-08-14 16:44:46 HKT; 10min ago Main PID: 13905 (etcd) CGroup: /system.slice/etcd.service └─13905 /usr/bin/etcd --name infra1 --cert-file=/etc/etcd/etcdSSL/etcd.pem --key-file=/etc/etcd/etcdSSL/etcd-key.pem --peer-cert-file=/etc/etcd/etcdSSL/etcd.pem Aug 14 16:54:14 server81 etcd[13905]: established a TCP streaming connection with peer 9281d0058fa51de0 (stream Message writer)Aug 14 16:54:14 server81 etcd[13905]: established a TCP streaming connection with peer 9281d0058fa51de0 (stream MsgApp v2 writer)Aug 14 16:54:14 server81 etcd[13905]: health check for peer 9281d0058fa51de0 could not connect: dial tcp 172.16.5.87:2380: getsockopt: connection refusedAug 14 16:54:15 server81 etcd[13905]: updating the cluster version from 3.0 to 3.2Aug 14 16:54:15 server81 etcd[13905]: updated the cluster version from 3.0 to 3.2Aug 14 16:54:15 server81 etcd[13905]: enabled capabilities for version 3.2Aug 14 16:54:20 server81 etcd[13905]: 8da195f9b25012c9 [term: 1972] received a MsgApp message with higher term from 9281d0058fa51de0 [term: 1973]Aug 14 16:54:20 server81 etcd[13905]: 8da195f9b25012c9 became follower at term 1973Aug 14 16:54:20 server81 etcd[13905]: raft.node: 8da195f9b25012c9 changed leader from 8da195f9b25012c9 to 9281d0058fa51de0 at term 1973Aug 14 16:54:20 server81 etcd[13905]: 8da195f9b25012c9 [logterm: 1973, index: 13, vote: 0] rejected MsgVote from 9281d0058fa51de0 [logterm: 1972, index: 12] at term 1973[root@server81 etcd]#

etcdctl \ --ca-file=/etc/etcd/etcdSSL/ca.pem \ --cert-file=/etc/etcd/etcdSSL/etcd.pem \ --key-file=/etc/etcd/etcdSSL/etcd-key.pem \ cluster-health

最后的总结

在这里已经部署好etcd集群服务了，那么在此基础上，我们下一篇章就开始部署kubernetes的服务部署了。

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。