This method has been tested working on a single master k8s, for multi-master clusters, may not work!

Kubernetes requires certs on each nodes/masters to validate each other’s integrity, if the cert ever gets expired, you’d see an error like this: Unable to connect to the server: x509: certificate has expired or is not yet valid..

To fix this cluster, we first need to verify the cert status by:

$ openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 123123123123123(0x123123123123)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Nov 16 16:58:58 2017 GMT
            Not After : Nov 16 16:58:58 2018 GMT
...

This tells you that it expires on 2018.

Manual Update

Now we need to login to master nodes to reissue new cert and its related config files:

# Delete old keys
rm /etc/kubernetes/pki/{apiserver*,front-proxy-client*}
kubeadm init phase certs all --apiserver-advertise-address <ext IP> --apiserver-cert-extra-sans <int IP>
cd /etc/kubernetes/
# Delete old config
rm {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf}
kubeadm init phase kubeconfig all
reboot

An example of full cert all cmd:

kubeadm init phase certs all --apiserver-advertise-address 172.26.x.x --apiserver-cert-extra-sans 172.16.x.x --apiserver-cert-extra-sans lb-apiserver.kubernetes.local --apiserver-cert-extra-sans xxx-master-0.k8s.xxx.tdlab.ca --apiserver-cert-extra-sans xxx-master-0 --apiserver-cert-extra-sans 10.233.0.1

10.233.0.1 is quite important here as it’s the internal vip for all services.

init cmd is used when you don’t have certs at all or want to reissue brand new cert.

After master comes back online, issue new node temp token for nodes to join: kubeadm token create.

Then on each node, delete old config and replace with kubeadm issued new configs:

mv /etc/kubernetes/manifests /etc/kubernetes/manifests.bak
rm -rf /etc/kubernetes/kubelet.conf 
rm -rf  /etc/kubernetes/bootstrap-kubelet.conf 
rm -rf /etc/kubernetes/pki/ca.crt 
service kubelet stop
kubeadm join --token=7z7kgy.bef6tsdpiyxo4xj --discovery-token-unsafe-skip-ca-verification <ext IP>:6443
mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests
reboot

Auto Update Cluster Cert

Create a shell script with X permission:

/usr/local/bin/k8s-certs-renew.sh
#!/bin/bash

echo "## Expiration before renewal ##"
/usr/local/bin/kubeadm certs check-expiration

echo "## Renewing certificates managed by kubeadm ##"
/usr/local/bin/kubeadm certs renew all

echo "## Restarting control plane pods managed by kubeadm ##"
/usr/bin/docker ps -af 'name=k8s_POD_(kube-apiserver|kube-controller-manager|kube-scheduler|etcd)-*' -q | /usr/bin/xargs /usr/bin/docker rm -f

echo "## Updating /root/.kube/config ##"
/usr/bin/cp /etc/kubernetes/admin.conf /root/.kube/config

echo "## Waiting for apiserver to be up again ##"
until printf "" 2>>/dev/null >>/dev/tcp/127.0.0.1/6443; do sleep 1; done

echo "## Expiration after renewal ##"
/usr/local/bin/kubeadm certs check-expiration

It basically check cert status and renew them then restart control plane docker containers and wait until API server back online. kubeadm certs renew all is different from init phase certs, it only renews existing cert, after renewal you’ll see the cert begin date got kept, that’s why it doesn’t need to input --apiserver-cert-extra-sans keys.

Then create a system service and its timer:

/etc/systemd/system/k8s-certs-renew.service
[Unit]
Description=Renew K8S control plane certificates

[Service]
Type=oneshot
ExecStart=/usr/local/bin/k8s-certs-renew.sh

/etc/systemd/system/k8s-certs-renew.timer
[Unit]
Description=Timer to renew K8S control plane certificates

[Timer]
OnCalendar=Mon *-*-1,2,3,4,5,6,7 03:10:00

[Install]
WantedBy=multi-user.target

systemctl enable k8s-certs-renew.timer
systemctl start k8s-certs-renew.timer

It calls above shell script on every month’s first monday.