keepalived+haproxy部署高可用k8s集群

keepalived+haproxy部署高可用k8s集群

部署规划

首先准备几台服务器，计划部署3台master，3台 利用keepalived和haproxy做高可用，为了节约服务器，我将keepalived和haproxy和master一起部署。服务器规划如下：

集群版本：1.18.15-0

一、 环境准备

1、修改主机名 hostnamectl set-hostname k8s-master hostnamectl set-hostname k8s-master02 hostnamectl set-hostname k8s-master03 2、修改hosts文件 vi /etc/hosts ## 添加内容 172.16.0.10 k8svip 172.16.0.100 k8s-master 172.16.0.200 k8s-master02 172.16.0.201 k8s-master03 3、关闭防火墙等 ## 配置内核参数，将桥接的IPv4流量传递到iptables的链 cat < /proc/sys/net/ipv4/ip_forward #时间同步(在3台机运行) yum install ntpdate -y && timedatectl set-timezone Asia/Shanghai && ntpdate time.windows.com

二、 所有master节点安装keepalive和haproxy

1、 安装keepelived和haproxy

yum install -y keepelived haproxy

2、配置文件准备

keepalived配置文件 2 配置 keepalived cat < /etc/keepalived/keepalived.conf ! /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script check_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state ${STATE} interface ${INTERFACE} virtual_router_id ${ROUTER_ID} priority ${PRIORITY} authentication { auth_type PASS auth_pass ${AUTH_PASS} } virtual_ipaddress { ${APISERVER_VIP} } track_script { check_apiserver } } EOF 在上面的文件中替换自己相应的内容： /etc/keepalived/check_apiserver.sh 定义检查脚本路径 ${STATE} 如果是主节点 则为MASTER 其他则为 BACKUP。我这里选择k8s-master01为MASTER；k8s-master02 、k8s-master03为BACKUP； ${INTERFACE} 是网络接口，即服务器网卡的，我的服务器均为eth0； ${ROUTER_ID} 这个值只要在keepalived集群中保持一致即可，我使用的是默认值51； ${PRIORITY} 优先级，在master上比在备份服务器上高就行了。我的master设为100，备份服务50； ${AUTH_PASS} 这个值只要在keepalived集群中保持一致即可； ${APISERVER_VIP} 就是VIP的地址，我的为：172.16.0.10。

3 配置 keepalived健康检查

在上面的配置中我们也配置健康检查的参数，比如检查间隔时间，权重等等。创建脚本

vi /etc/keepalived/check_apiserver.sh ### 添加内容 #!/bin/sh errorExit() { echo "*** $*" 1>&2 exit 1 } curl --silent --max-time 2 --insecure -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/" if ip addr | grep -q ${APISERVER_VIP}; then curl --silent --max-time 2 --insecure -o /dev/null || errorExit "Error GET https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/" fi ${APISERVER_VIP} 就是VIP的地址，172.16.0.10； ${APISERVER_DEST_PORT} 这个是定义API Server交互的负载均衡端口，其实就是HAProxy绑定前端负载均衡的端口号，因为HAProxy和k8s一起部署，这里做一个区分，我使用了8443，这个下面会说到。

4、配置haproxy

# /etc/haproxy/haproxy.cfg #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log /dev/log local0 log /dev/log local1 notice daemon #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 1 timeout 10s timeout queue 20s timeout connect 5s timeout client 20s timeout server 20s timeout 10s timeout check 10s #--------------------------------------------------------------------- # apiserver frontend which proxys to the masters #--------------------------------------------------------------------- frontend apiserver bind *:${APISERVER_DEST_PORT} mode tcp option tcplog default_backend apiserver #--------------------------------------------------------------------- # round robin balancing for apiserver #--------------------------------------------------------------------- backend apiserver option GET /healthz expect status 200 mode tcp option ssl-hello-chk balance roundrobin server ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check 上面的配置只需要修改为如下内容： ${APISERVER_DEST_PORT} 这个值是定义前端负载均衡端口，同上面的健康检查脚本里面的值一样，我这里使用8443； ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} 这里定义的是后端节点的IP:Prot，比如我的配置是： ### server ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check server k8s-master 172.16.0.100:6443 check server k8s-master02 172.16.0.200:6443 check server k8s-master03 172.16.0.201:6443 check

5、启动服务并设置开机自启

systemctl enable haproxy --now systemctl enable keepalived --now 因为后端服务还未启动所以无法探测到服务，请忽略下面的提示： [root@k8s-master ~]# Broadcast message from systemd-journald@k8s-master (Wed 2021-04-21 14:39:55 CST): haproxy[4758]: backend apiserver has no server available! Broadcast message from systemd-journald@k8s-master (Wed 2021-04-21 14:39:55 CST): haproxy[4758]: backend apiserver has no server available! Message from syslogd@k8s-master at Apr 21 14:39:55 ... haproxy[4758]:backend apiserver has no server available! Message from syslogd@k8s-master at Apr 21 14:39:55 ... haproxy[4758]:backend apiserver has no server available!

四、安装docker

#1 安装必要的一些系统工具 sudo yum install -y yum-utils device-mapper-persistent-data lvm2 # 2: 添加软件源信息 sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 3: 更新并安装Docker-CE sudo yum makecache fast sudo yum -y install docker-ce # 4: 开启Docker服务 sudo systemctl start docker && systemctl enable docker # 注意： # 官方软件源默认启用了最新的软件，您可以通过编辑软件源的方式获取各个版本的软件包。例如官方并没有将测试版本的软件源置为可用，您可以通过以下方式开启。同理可以开启各种测试版本等。 # vim /etc/yum.repos.d/docker-ee.repo # 将[docker-ce-test]下方的enabled=0修改为enabled=1 # # 安装指定版本的Docker-CE: # Step 1: 查找Docker-CE的版本: # yum list docker-ce.x86_64 --showduplicates | sort -r # Loading mirror speeds from cached hostfile # Loaded plugins: branch, fastestmirror, langpacks # docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable # docker-ce.x86_64 17.03.1.ce-1.el7.centos @docker-ce-stable # docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable # Available Packages # Step2: 安装指定版本的Docker-CE: (VERSION例如上面的17.03.0.ce.1-1.el7.centos) # sudo yum -y install docker-ce-[VERSION] # docker镜像加速，"https://s2q9fn53.mirror.aliyuncs.com"这个地址建议自己登陆阿里云，在容器镜像服务中找到。 # 可以通过修改daemon配置文件/etc/docker/daemon.json来使用加速器 sudo mkdir -p /etc/docker sudo cat >> /etc/docker/daemon.json <<-'EOF' { "exec-opts": ["native.cgroupdriver=systemd"], "registry-mirrors": ["https://s2q9fn53.mirror.aliyuncs.com"] } EOF sudo systemctl daemon-reload && sudo systemctl restart docker

五、安装kubelet、kubeadm、kubectl

kubeadm：用来初始化集群的指令。 kubelet：在集群中的每个节点上用来启动 pod 和容器等。 kubectl：用来与集群通信的命令行工具。

kubeadm 不能 帮您安装或者管理 kubelet 或 kubectl，所以您需要确保它们与通过 kubeadm 安装的控制平面的版本相匹配。 如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。 然而，控制平面与 kubelet 间的相差一个次要版本不一致是支持的，但 kubelet 的版本不可以超过 API 服务器的版本。 例如，1.7.0 版本的 kubelet 可以完全兼容 1.8.0 版本的 API 服务器，反之则不可以。

#添加kubernetes阿里YUM源 cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF yum install -y kubelet-1.18.15-0 kubeadm-1.18.15-0 kubectl-1.18.15-0 && systemctl enable kubelet && systemctl start kubelet

六、初始化master集群

#注意，kubeadm init 前,先准备k8s运行所需的容器 #可查询到kubernetes所需镜像 [root@k8s-master ~]# kubeadm config images list I0421 14:58:20.854664 5292 version.go:252] remote version is much newer: v1.21.0; falling back to: stable-1.18 W0421 14:58:23.294529 5292 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] #下面显示的就是这个版本所需要下载的镜像 k8s.gcr.io/kube-apiserver:v1.18.18 k8s.gcr.io/kube-controller-manager:v1.18.18 k8s.gcr.io/kube-scheduler:v1.18.18 k8s.gcr.io/kube-proxy:v1.18.18 k8s.gcr.io/pause:3.2 k8s.gcr.io/etcd:3.4.3-0 k8s.gcr.io/coredns:1.6.7 #写了个sh脚本，把所需的镜像拉下来 cat >> alik8simages.sh << EOF #!/bin/bash list='kube-apiserver:v1.18.18 kube-controller-manager:v1.18.18 kube-scheduler:v1.18.18 kube-proxy:v1.18.18 pause:3.2 etcd:3.4.13-0 coredns:coredns:1.6.7' for item in \$list do docker pull registry.aliyuncs.com/google_containers/\$item && docker tag registry.aliyuncs.com/google_containers/\$item k8s.gcr.io/\$item && docker rmi registry.aliyuncs.com/google_containers/\$item done EOF #运行脚本下载，可能会失败，失败了就是网络问题多执行即便就行 bash alik8simages.sh #检查镜像 [root@k8s-master ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.18.18 8bd0db6f4d0a 6 days ago 117MB k8s.gcr.io/kube-apiserver v1.18.18 5745154baa89 6 days ago 173MB k8s.gcr.io/kube-controller-manager v1.18.18 9fb627f53264 6 days ago 162MB k8s.gcr.io/kube-scheduler v1.18.18 fe100f0c6984 6 days ago 96.1MB k8s.gcr.io/etcd 3.4.13-0 0369cf4303ff 7 months ago 253MB k8s.gcr.io/pause 3.2 80d28bedfe5d 14 months ago 683kB #k8s主节点初始化 注意一下操作仅需要在master01节点执行 1、生成初始化配置文件 [root@k8s-master ~]# kubeadm config print init-defaults >kubeadm-config.yml 2、修改配置文件： [root@k8s-master ~]# cat kubeadm-config.yml apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: “advertiseAddress: 172.16.0.100” #修改为本地监听地址 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: k8s-master taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: "172.16.0.10:8443" #修改为负载均衡地址和端口 controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.18.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {} #初始化开始 [root@k8s-master ~]# kubeadm init --config kubeadm-config.yml W0421 15:13:55.397335 6965 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.0 [preflight] Running pre-flight checks [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service' [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 19.03 [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.18.0: output: Error response from daemon: Get net/request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ... 注意kubeadm config images list 有时候这个命令查的版本不是真正需要的镜像就会报上面的错误这时候就需要重新下载报错提示v1.18.0的镜像，而后重试初始化

保存好如下内容

#该内容是其他master加入需要执行的命令 kubeadm join 172.16.0.10:8443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:bfdc983afbdbe560a0ebf2c1bf1007c22b04a76a441fb49ad2e955ad2d588977 \ --control-plane #该内容是node节点加入需要执行的命令 kubeadm join 172.16.0.10:8443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:bfdc983afbdbe560a0ebf2c1bf1007c22b04a76a441fb49ad2e955ad2d588977

9、配置API所需的配置文件

mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config

10、安装集群网络组件flannel

[root@k8s-node01 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 回显信息 podsecuritypolicy.extensions/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.extensions/kube-flannel-ds-amd64 created daemonset.extensions/kube-flannel-ds-arm64 created daemonset.extensions/kube-flannel-ds-arm created daemonset.extensions/kube-flannel-ds-ppc64le created daemonset.extensions/kube-flannel-ds-s390x created 或安装cannl ]# kubectl apply -f https://docs.projectcalico.org/v3.manifests/canal.yaml configmap/canal-config created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/canal-flannel created clusterrolebinding.rbac.authorization.k8s.io/canal-calico created daemonset.apps/canal created serviceaccount/canal created

master02 master03加入集群

1、创建pki和etcd目录 [root@k8s-master03 kubernetes]# mkdir -p /etc/kubernetes/pki/etcd 2、从master01 拷贝证书至两个节点 使用该脚本需要提前做好免密登入 [root@k8s-master ~]# cat cpkey.sh USER=root # 账号 CONTROL_PLANE_IPS="172.16.0.200 172.16.0.201" #节点IP dir=/etc/kubernetes/pki/ for host in ${CONTROL_PLANE_IPS}; do scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:${dir} scp /etc/kubernetes/pki/ca.key "${USER}"@$host:${dir} scp /etc/kubernetes/pki/sa.key "${USER}"@$host:${dir} scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:${dir} scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:${dir} scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:${dir} scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:${dir}etcd #如果您使用的是外部etcd，请引用此行 scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:${dir}etcd done 3、#master02 master03加入集群 [root@k8s-master03 kubernetes]#kubeadm join 172.16.0.10:8443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:bfdc983afbdbe560a0ebf2c1bf1007c22b04a76a441fb49ad2e955ad2d588977 4、配置API所需的配置文件 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config

最后查看集群状态

[root@k8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready master 58m v1.18.15 k8s-master02 Ready master 15m v1.18.15 k8s-master03 Ready master 5m39s v1.18.15

遇到的问题

该问题是因为没有将主master的证书拷贝过来导致无法连接api-server容器

[root@k8s-master02 ~]# kubeadm join 172.16.0.10:8443 --token abcdef.0123456789abcdef \ > --discovery-token-ca-cert-hash sha256:bfdc983afbdbe560a0ebf2c1bf1007c22b04a76a441fb49ad2e955ad2d588977 \ > --control-plane [preflight] Running pre-flight checks [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service' [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 19.03 [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' error execution phase preflight: One or more conditions for hosting a new control plane instance is not satisfied. failure loading certificate for CA: couldn't load the certificate file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory Please ensure that: * The cluster has a stable controlPlaneEndpoint address. * The certificates that must be shared among control plane instances are provided. To see the stack trace of this error execute with --v=5 or higher

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。