如果k8s证书过期，会导致服务连接不可用，需要进行k8s证书更新操作
k8s证书过期检查
通过 kubeadm 安装的 k8s 集群，证书有效期为 1年，一年过期后，会导致 api service 不可用，使用过程中会出现报错：
1
| x509: certificate has expired or is not yet valid.
|
通过实测，记录两种切实可用的方案：
- 手动更新证书（证书已过期也可用）；
- 源码编译kubeadm，使证书时间更长（集群证书已过期也可用）
系统版本
1
2
| System: CentOS Linux release 7.9.2009 (Core)
kernel: 3.10.0-1160.el7.x86_64
|
docker版本
1
2
| Server: 18.06.3-ce
Client: 18.06.3-ce
|
Kubernetes 版本
在 master 上使用如下命令查看证书过期时间：
1
| kubeadm alpha certs check-expiration
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| root@k8s-master(10.0.0.11)~>kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
W0818 15:09:56.236357 4122 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 18, 2022 07:04 UTC <invalid> no
apiserver Aug 18, 2022 07:04 UTC <invalid> ca no
apiserver-etcd-client Aug 18, 2022 07:04 UTC <invalid> etcd-ca no
apiserver-kubelet-client Aug 18, 2022 07:04 UTC <invalid> ca no
controller-manager.conf Aug 18, 2022 07:04 UTC <invalid> no
etcd-healthcheck-client Aug 18, 2022 07:04 UTC <invalid> etcd-ca no
etcd-peer Aug 18, 2022 07:04 UTC <invalid> etcd-ca no
etcd-server Aug 18, 2022 07:04 UTC <invalid> etcd-ca no
front-proxy-client Aug 18, 2022 07:04 UTC <invalid> front-proxy-ca no
scheduler.conf Aug 18, 2022 07:04 UTC <invalid> no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 16, 2031 07:04 UTC 8y no
etcd-ca Aug 16, 2031 07:04 UTC 8y no
front-proxy-ca Aug 16, 2031 07:04 UTC 8y no
|
该命令显示 /etc/kubernetes/pki 文件夹中的客户端证书以及 kubeadm 使用的 KUBECONFIG 文件中嵌入的客户端证书的到期时间/剩余时间。
也可通过如下命令查看证书详细时间：
1
| openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | egrep Not
|
1
2
3
| openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | egrep Not
Not Before: Aug 18 07:24:06 2021 GMT
Not After : Aug 18 07:24:06 2022 GMT
|
执行和 api service的交互报错如下：
1
2
| root@k8s-master(10.0.0.11)~>kubectl get po
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-08-18T15:10:07+08:00 is after 2022-08-18T07:04:33Z
|
以上错误表示证书到期，可用如下两种方式进行更新。
手动更新证书
1. 备份
注意：做任何改动之前都需要进行源文件备份
1
2
3
| mkdir -pv /etc/kubernetes-bak
cp -r /etc/kubernetes/pki /etc/kubernetes-bak/
cp /etc/kubernetes/*.conf /etc/kubernetes-bak/
|
etcd 数据目录备份：
1
| cp -ar /var/lib/etcd /var/lib/etcd-bak
|
2. 执行更新证书
1
| kubeadm alpha certs renew all
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
W0818 15:46:40.020407 3775 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
|
通过上面的命令证书就一键更新完成了，这个时候查看上面的证书可以看到过期时间已经是一年后的时间了：
1
| kubeadm alpha certs check-expiration
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| root@k8s-master(10.0.0.11)~>kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
W0818 15:47:22.712349 3834 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 18, 2023 07:46 UTC 364d no
apiserver Aug 18, 2023 07:46 UTC 364d ca no
apiserver-etcd-client Aug 18, 2023 07:46 UTC 364d etcd-ca no
apiserver-kubelet-client Aug 18, 2023 07:46 UTC 364d ca no
controller-manager.conf Aug 18, 2023 07:46 UTC 364d no
etcd-healthcheck-client Aug 18, 2023 07:46 UTC 364d etcd-ca no
etcd-peer Aug 18, 2023 07:46 UTC 364d etcd-ca no
etcd-server Aug 18, 2023 07:46 UTC 364d etcd-ca no
front-proxy-client Aug 18, 2023 07:46 UTC 364d front-proxy-ca no
scheduler.conf Aug 18, 2023 07:46 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 16, 2031 07:39 UTC 8y no
etcd-ca Aug 16, 2031 07:39 UTC 8y no
front-proxy-ca Aug 16, 2031 07:39 UTC 8y no
|
3. 更新 kubeconfig 文件
1
| kubeadm init phase kubeconfig all
|
1
2
3
4
5
6
7
8
9
| root@k8s-master(10.0.0.11)~>kubeadm init phase kubeconfig all
I0818 15:48:50.617013 3929 version.go:252] remote version is much newer: v1.24.4; falling back to: stable-1.19
W0818 15:48:52.891141 3929 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
|
4. 覆盖原admin文件
1
2
3
| mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
|
5. 重启容器
重启 kube-apiserver、kube-controller、kube-scheduler、etcd 这4个容器即可
1
2
3
4
5
| docker restart `docker ps | grep etcd | awk '{print $1}'`
docker restart `docker ps | grep kube-apiserver | awk '{print $1}'`
docker restart `docker ps | grep kube-controller | awk '{print $1}'`
docker restart `docker ps | grep kube-scheduler | awk '{print $1}'`
|
6. 验证是否认证成功
查看pod集群状态，检查刚刚重启的status是否为Running(一般会等待2分钟左右)
1
| kubectl get po -A -o wide
|
查看当前集群的证书时间，RESIDUAL TIME，为364d，续期一年。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| root@k8s-master(10.0.0.11)~>kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 18, 2023 07:46 UTC 364d no
apiserver Aug 18, 2023 07:46 UTC 364d ca no
apiserver-etcd-client Aug 18, 2023 07:46 UTC 364d etcd-ca no
apiserver-kubelet-client Aug 18, 2023 07:46 UTC 364d ca no
controller-manager.conf Aug 18, 2023 07:46 UTC 364d no
etcd-healthcheck-client Aug 18, 2023 07:46 UTC 364d etcd-ca no
etcd-peer Aug 18, 2023 07:46 UTC 364d etcd-ca no
etcd-server Aug 18, 2023 07:46 UTC 364d etcd-ca no
front-proxy-client Aug 18, 2023 07:46 UTC 364d front-proxy-ca no
scheduler.conf Aug 18, 2023 07:46 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 16, 2031 07:39 UTC 8y no
etcd-ca Aug 16, 2031 07:39 UTC 8y no
front-proxy-ca Aug 16, 2031 07:39 UTC 8y no
|
到此，通过手动的方式更新证书完成。
源码编译kubeadm
可以在初始化集群之前重新编译 kubeadm，证书有限期可自行指定。
新建集群如下操作：
1. 获取源码
如果集群已经初始化完成，请下载对应版本的源代码进行操作。
我这里的版本为 v1.19.7 去 github 下载对应版本的源代码进行修改编译。
注意：版本一定要一致！
1
2
3
4
| wget https://github.com/kubernetes/kubernetes/archive/v1.19.7.tar.gz
tar -zxvf v1.19.7.tar.gz
mv kubernetes-1.19.7 kubernetes
cd kubernetes
|
2. 修改CA为100年
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| >vim ./staging/src/k8s.io/client-go/util/cert/cert.go
...
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
now := time.Now()
tmpl := x509.Certificate{
SerialNumber: new(big.Int).SetInt64(0),
Subject: pkix.Name{
CommonName: cfg.CommonName,
Organization: cfg.Organization,
},
NotBefore: now.UTC(),
NotAfter: now.Add(duration365d * 100).UTC(), # 这里10修改为100
...
>vim ./cmd/kubeadm/app/constants/constants.go
...
const (
// KubernetesDir is the directory Kubernetes owns for storing various configuration files
KubernetesDir = "/etc/kubernetes"
// ManifestsSubDirName defines directory name to store manifests
ManifestsSubDirName = "manifests"
// TempDirForKubeadm defines temporary directory for kubeadm
// should be joined with KubernetesDir.
TempDirForKubeadm = "tmp"
// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
CertificateValidity = time.Hour * 24 * 365 * 100 # 这里乘以 100 年
...
|
3. 本地编译
1
| yum install -y gcc make jq rsync
|
4. 安装go环境
查看 kube-cross 的 TAG 版本号
1
2
| cat ./build/build-image/cross/VERSION
v1.15.5-1
|
下载go，上传到 master节点
1
2
3
4
5
6
7
8
9
10
11
| https://gomirrors.org/dl/go/go1.15.1.linux-amd64.tar.gz
>tar xf go1.15.Linux-amd64.tar.gz -C /usr/local/
# 编辑/etc/profile文件添加如下：
>vim /etc/profile
...
export GOROOT=/usr/local/go
export GOPATH=/usr/local/gopath
export PATH=$PATH:$GOROOT/bin
>source /etc/profile
|
验证：
1
2
| >go version
go version go1.15 linux/amd64
|
5. 编译kubeadm
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| # 编译kubeadm, 这里主要编译kubeadm 即可
make all WHAT=cmd/kubeadm GOFLAGS=-v
# 编译kubelet
# make all WHAT=cmd/kubelet GOFLAGS=-v
# 编译kubectl
# make all WHAT=cmd/kubectl GOFLAGS=-v
#编译完产物在 _output/bin/kubeadm 目录下，
#其中bin是使用了软连接
#真实路径是_output/local/bin/Linux/amd64/kubeadm
mv /usr/bin/kubeadm /usr/bin/kubeadm_backup
cp _output/local/bin/Linux/amd64/kubeadm /usr/bin/kubeadm
chmod +x /usr/bin/kubeadm
|
6. 更新证书
查看证书到期时间：
1
| kubeadm alpha certs check-expiration
|
续订全部证书：
1
| kubeadm alpha certs renew all
|
再次查看证书到期时间：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| root@k8s-master(10.0.0.11)~>kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 25, 2121 10:00 UTC 99y no
apiserver Jul 25, 2121 10:00 UTC 99y ca no
apiserver-etcd-client Jul 25, 2121 10:00 UTC 99y etcd-ca no
apiserver-kubelet-client Jul 25, 2121 10:00 UTC 99y ca no
controller-manager.conf Jul 25, 2121 10:00 UTC 99y no
etcd-healthcheck-client Jul 25, 2121 10:00 UTC 99y etcd-ca no
etcd-peer Jul 25, 2121 10:00 UTC 99y etcd-ca no
etcd-server Jul 25, 2121 10:00 UTC 99y etcd-ca no
front-proxy-client Jul 25, 2121 10:00 UTC 99y front-proxy-ca no
scheduler.conf Jul 25, 2121 10:00 UTC 99y no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 25, 2121 10:00 UTC 99y no
etcd-ca Jul 25, 2121 10:00 UTC 99y no
front-proxy-ca Jul 25, 2121 10:00 UTC 99y no
|
如果集群证书已经过期，在上面的基础继续如下步骤：
1
2
3
4
5
| docker restart `docker ps | grep etcd | awk '{print $1}'`
docker restart `docker ps | grep kube-apiserver | awk '{print $1}'`
docker restart `docker ps | grep kube-controller | awk '{print $1}'`
docker restart `docker ps | grep kube-scheduler | awk '{print $1}'`
|
更新 kubeconfig 文件
1
| kubeadm init phase kubeconfig all
|
1
2
3
4
5
6
7
8
9
| root@k8s-master(10.0.0.11)~>kubeadm init phase kubeconfig all
I0818 15:48:50.617013 3929 version.go:252] remote version is much newer: v1.24.4; falling back to: stable-1.19
W0818 15:48:52.891141 3929 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
|
覆盖原admin文件
1
2
3
| mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
|
到此，编译更新证书完成。
操作命令备注
背景：K8S 各个组件需要与 api-server 进行通信，通信使用的证书都存放在 /etc/kubernetes/pki 路径下，kubeadm 生成的证书默认有效为 1 年，因此需要定时更新证书，否则证书到期会导致整个集群不可用。 自动更新证书，Kubenetes 在升级控制面板相关组件时会主动更新证书，因此如果保证 Kubernetes 能够定期（一年以内）升级的话，证书会自动更新。
环境说明：k8s 1.18.10
以下步骤，在所有 master 节点执行
具体操作步骤
1）检查证书是否过期，以下两个命令均可
1
2
| kubeadm alpha certs check-expiration
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
|
2）证书、配置文件等备份
1
| cp -rp /etc/kubernetes /etc/kubernetes.bak
|
3）重新生成证书
1
| kubeadm alpha certs renew all
|
4）重新生成配置文件
1
2
| rm -f /etc/kubernetes/*.conf
kubeadm init phase kubeconfig all
|
5）重启 kubelet、apiserver、scheduler、controller-manager、etcd
1
2
| systemctl restart kubelet
docker ps | egrep "k8s_kube-apiserver|k8s_kube-scheduler|k8s_kube-controller-manager|k8s_etcd" | awk '{print $1}' | xargs docker restart
|
6）更新 admin 配置
1
| cp /etc/kubernetes/admin.conf ~/.kube/config
|
7）检查证书过期时间
1
| kubeadm alpha certs check-expiration
|
如果重新生成过程中有错误，就把旧文件删除