This method has been tested working on a single master k8s, for multi-master clusters, may not work!

Kubernetes requires certs on each nodes/masters to validate each other’s integrity, if the cert ever gets expired, you’d see an error like this: Unable to connect to the server: x509: certificate has expired or is not yet valid..

To fix this cluster, we first need to verify the cert status by:

$ openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt
        Version: 3 (0x2)
        Serial Number: 123123123123123(0x123123123123)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
            Not Before: Nov 16 16:58:58 2017 GMT
            Not After : Nov 16 16:58:58 2018 GMT

This tells you that it expires on 2018.

Now we need to login to master nodes to reissue new cert and its related config files:

# Delete old keys
rm /etc/kubernetes/pki/{apiserver*,front-proxy-client*}
kubeadm init phase certs all --apiserver-advertise-address <ext IP> --apiserver-cert-extra-sans <int IP>
cd /etc/kubernetes/
# Delete old config
rm {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf}
kubeadm init phase kubeconfig all

An example of full cert all cmd kubeadm init phase certs all --apiserver-advertise-address 172.26.x.x --apiserver-cert-extra-sans 172.16.x.x --apiserver-cert-extra-sans lb-apiserver.kubernetes.local --apiserver-cert-extra-sans --apiserver-cert-extra-sans xxx-master-0 --apiserver-cert-extra-sans, is quite important here as it’s the internal vip for all services.

After master comes back online, issue new node temp token for nodes to join: kubeadm token create.

Then on each node, delete old config and replace with kubeadm issued new configs:

mv /etc/kubernetes/manifests /etc/kubernetes/manifests.bak
rm -rf /etc/kubernetes/kubelet.conf 
rm -rf  /etc/kubernetes/bootstrap-kubelet.conf 
rm -rf /etc/kubernetes/pki/ca.crt 
service kubelet stop
kubeadm join --token=7z7kgy.bef6tsdpiyxo4xj --discovery-token-unsafe-skip-ca-verification <ext IP>:6443
mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests