Possible Issues and Solutions in Kubernetes, Docker, and Containerd

sudo swapoff -a sudo sed -i '/ swap / s/^/#/' /etc/fstab

sudo reboot

kubeadm reset kubeadm init --ignore-preflight-errors all

deleting namespace stuck at "Terminating" state

kubectl get namespace "<NAMESPACE>" -o json | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" | kubectl replace --raw /api/v1/namespaces/<NAMESPACE>/finalize -f -

$ sudo vi /etc/docker/daemon.json

"insecure-registries" : ["hub.docker.com:443", "registry-1.docker.io:443", "quay.io"]

sudo systemctl daemon-reload
sudo systemctl restart docker

#It is checked with the following.
docker info

cp ssl.crt /usr/local/share/ca-certificates/
update-ca-certificates
service docker restart

#Centos 7
sudo cp -p ssl.crt /etc/pki/ca-trust/source
sudo cp ssl.crt /etc/pki/ca-trust/source/anchors/myregistrydomain.com.crt

sudo update-ca-trust extract
sudo systemctl daemon-reload
sudo systemctl restart docker

$ sudo vi /etc/docker/daemon.json

{

"data-root":"/docker-data",

"insecure-registries":["nexusdocker.institutionaddress.com.tr"],

"registry-mirrors":["https://nexusdocker.institutionaddress.com.tr"],

"exec-opts": ["native.cgroupdriver=systemd"],

"log-driver": "json-file",

"log-opts": { "max-size": "100m" },

"storage-driver": "overlay2"

}

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

kubectl get pods dnsutils

kubectl exec -i -t dnsutils -- nslookup kubernetes.default

If the result is as below everything is correct.

Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      kubernetes.default
Address 1: 10.0.0.1

If the result is as follows, there is an error and the following steps need to be checked.Server: 10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1

Check the Resolv.conf file.

$ kubectl exec -ti dnsutils -- cat /etc/resolv.conf

(correct)

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local institution.gov.tr
options ndots:5

(false)

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

kubectl rollout restart -n kube-system deployment/coredns

In a client, it was resolved by adding the institution's domain address to the /etc/resolv.conf file.

search institution.gov.tr

On Ubuntu servers, changes made to the dns server may not always be reflected in resolv.conf or may be skipped. Since Kubernetes by default looks at the cat /etc/resolv.conf file on the server after its own internal dns, you should make sure that this file is correct.

On all server:

sudo rm /etc/resolv.conf

sudo ln -s /run/systemd/resolve/resolv.conf  /etc/resolv.conf

sudo systemctl restart systemd-resolved

ls -l  /etc/resolv.conf

cat  /etc/resolv.conf

Only on the master server:

kubectl -n kube-system rollout restart deployment coredns

(Alternative solutions: https://github.com/kubernetes/kubernetes/issues/54918)

$ sudo mkdir -p /etc/cni/net.d

$ sudo vi /etc/cni/net.d/10-flannel.conflist

#add the below.

----------

------------

sudo chmod -Rf 777 /etc/cni /etc/cni/*

sudo chown -Rf apinizer:apinizer /etc/cni /etc/cni/*

sudo systemctl daemon-reload

sudo systemctl restart kubelet

#Check if there is still a pod that cannot take an image:

kubectl get pods -n kube-system

describe pod podAdi -n kube-system

Unable to connect to the server: x509: certificate has expired or is not yet valid

#These operations should be done on all master servers.

sudo kubeadm alpha certs check-expiration 
sudo kubeadm alpha certs renew all 

mkdir -p $HOME/.kube 
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 
sudo chown $(id -u):$(id -g) $HOME/.kube/config

#all the nodes
sudo reboot -i

#further readings:
https://serverfault.com/questions/1065444/how-can-i-find-which-kubernetes-certificate-has-expired)
https://www.oak-tree.tech/blog/k8s-cert-yearly-renewwal

That problem can occur from the reasons below:

• If the disk capacity extended, swap migth have been opened.
• The user may not have authorization.
• You may not be on the Master Kubernetes Server.

sudo swapoff -a

sudo vi /etc/fstab (swap line must be commented out or deleted)

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

sudo reboot    (optional) 

#Existing configs and certificates are backed up and operations are performed

cd /etc/kubernetes/pki/
sudo mkdir /tmp/backup | sudo mkdir /tmp/backup2
sudo mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} /tmp/backup/

sudo kubeadm init phase certs all --apiserver-advertise-address <MasterIP>
cd /etc/kubernetes/
sudo mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/backup2

sudo kubeadm init phase kubeconfig all
sudo systemctl restart docker %% sudo systemctl restart containerd && sudo systemctl restart kubelet

We provide the solution with the -skip-verify parameter.

For example, the command to include it in the "k8s.io" namespace:

ctr --namespace k8s.io images pull xxx.harbor.com/apinizercloud/managerxxxx -skip-verify

When a node in Kubernetes shuts down unexpectedly (Non-Graceful Shutdown), the Kubernetes Master detects this situation and takes necessary actions. However, this detection process may be delayed because it depends on the timeout parameters of the system.