Kubernetes, Docker and Containerd Possible Issues and Solutions

Reason/Why: Flannel files and settings used for pod overlay network remain on the server and need to be manually deleted.

Solution: On the control-plane Kubernetes node, remove the relevant server from the cluster with “kubectl delete node <NODE_NAME>”, then clean the cluster settings on the server to be disconnected with “sudo kubeadm reset” command. Then perform the following operations:

systemctl stop kubelet && systemctl stop containerd
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1
systemctl restart containerd && systemctl restart kubelet

Reason/Why: With the release of RHEL 8 and CentOS 8, the docker package was removed from default package repositories, docker was replaced with podman and buildah. RedHat decided not to provide official support for Docker. Therefore, these packages prevent Docker installation.

Solution:

yum remove podman* -y
yum remove buildah* -y

Reason/Why: “swap” and “selinux” which are usually active in Linux operating systems should be disabled.

Solution:

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab
sudo reboot
kubeadm reset
kubeadm init --ignore-preflight-errors all

Reason/Why: Namespace deletion operation may get stuck due to finalizers.

Solution:

kubectl get namespace "<NAMESPACE>" -o json | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" | kubectl replace --raw /api/v1/namespaces/<NAMESPACE>/finalize -f -

Reason/Why: If the relevant organization is not using HTTPS, the following line is added to Docker’s daemon file. This operation is repeated for all nodes using Docker.

Solution (For organizations not using HTTPS):

sudo vi /etc/docker/daemon.json

{
  "insecure-registries": ["hub.docker.com:443", "registry-1.docker.io:443", "quay.io"]
}

sudo systemctl daemon-reload
sudo systemctl restart docker
# Check with the following
docker info

Solution (For organizations using HTTPS): If the relevant organization is using HTTPS, SSL certificate (“crt”) from the relevant organization needs to be added to servers.

# For Ubuntu/Debian
cp ssl.crt /usr/local/share/ca-certificates/
update-ca-certificates
service docker restart

# For CentOS 7
sudo cp -p ssl.crt /etc/pki/ca-trust/source
sudo cp ssl.crt /etc/pki/ca-trust/source/anchors/myregistrydomain.com.crt
sudo update-ca-trust extract
sudo systemctl daemon-reload
sudo systemctl restart docker

Reason/Why: If the relevant organization uses Nexus proxy, Docker servers are redirected to this address.

Solution:

sudo vi /etc/docker/daemon.json

{
  "data-root": "/docker-data",
  "insecure-registries": ["nexusdocker.kurumunadresi.com.tr"],
  "registry-mirrors": ["https://nexusdocker.kurumunadresi.com.tr"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}

Reason/Why: Node may stay in Ready, SchedulingDisabled state.

Test:

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl get pods dnsutils
kubectl exec -i -t dnsutils -- nslookup kubernetes.default

If we get the following result, everything is correct:

Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      kubernetes.default
Address 1: 10.0.0.1

If we get the following result, there is an error and the following steps need to be checked:

Server: 10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1

Take a look at the resolv.conf file:

kubectl exec -ti dnsutils -- cat /etc/resolv.conf

(Correct)

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local kurum.gov.tr
options ndots:5

(Incorrect)

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Solution: It was resolved by adding the organization’s domain address to the /etc/resolv.conf file in a customer.

# search kurum.gov.tr is added
kubectl rollout restart -n kube-system deployment/coredns

Reason/Why: On Ubuntu operating system servers, changes made regarding DNS server may not always reflect in resolv.conf or may be skipped. Since Kubernetes by default looks at the cat /etc/resolv.conf file on the server after its internal DNS, this file must be ensured to be correct.

Solution:

On all servers:

sudo rm /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
sudo systemctl restart systemd-resolved
ls -l /etc/resolv.conf
cat /etc/resolv.conf

Only on master server:

kubectl -n kube-system rollout restart deployment coredns

Reason/Why: Firewall performs SSL inspection and adds its own certificate.

Solution: docker.io should be added to “SSL inspection exception” on the firewall.

Reason/Why: kube-flannel on Master cannot create the necessary folder and files somehow.

Solution: (Alternative solutions are also available: GitHub Issue #54918)

sudo mkdir -p /etc/cni/net.d
sudo vi /etc/cni/net.d/10-flannel.conflist

The following content is added:

{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

sudo chmod -Rf 777 /etc/cni /etc/cni/*
sudo chown -Rf apinizer:apinizer /etc/cni /etc/cni/*
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Check if there are pods that still cannot pull images:
kubectl get pods -n kube-system
kubectl describe pod <podAdi> -n kube-system

Reason/Why: Unable to connect to the server: x509: certificate has expired or is not yet valid.

Solution: These operations must be performed on all master servers.

sudo kubeadm certs check-expiration
sudo kubeadm certs renew all

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# All master/control-plane nodes
sudo reboot

# For more information:
# https://serverfault.com/questions/1065444/how-can-i-find-which-kubernetes-certificate-has-expired
# https://www.oak-tree.tech/blog/k8s-cert-yearly-renewwal

Reason/Why: The above problem can occur due to any of the following reasons:

• Swap may need to be closed again as it can be opened when disk is added.
• User may not have permissions.
• We may not be on the master server.

Solution:

sudo swapoff -a
sudo vi /etc/fstab  # swap line will be closed or deleted
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
sudo reboot  # optional

Reason/Why: While this problem has various causes, if the error says that any .conf file such as /etc/kubernetes/bootstrap-kubelet.conf cannot be found, all configs can be recreated from scratch by applying the following operations.

Solution (Master Node): Operations are performed by backing up existing configs and certificates.

cd /etc/kubernetes/pki/
sudo mkdir /tmp/backup
sudo mkdir /tmp/backup2
sudo mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} /tmp/backup/
sudo kubeadm init phase certs all --apiserver-advertise-address <MASTER_NODE_IP>

cd /etc/kubernetes/
sudo mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/backup2
sudo kubeadm init phase kubeconfig all
sudo systemctl restart docker && sudo systemctl restart containerd && sudo systemctl restart kubelet

Solution (Worker Node): If the error of not finding the /etc/kubernetes/bootstrap-kubelet.conf file occurs on Worker Nodes as well, removing the node from the cluster and adding it again will solve the problem.

Commands to be run on master node:

# First, the problematic worker node is removed from the cluster
kubectl delete node <WORKER_NODE_NAME>

# Then a new join token is created
sudo kubeadm token create --print-join-command

Commands to be run on worker node:

# Kubernetes configuration is reset
sudo kubeadm reset

# The join command received from master is executed
sudo kubeadm join <MASTER_IP>:<PORT> --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>

Reason/Why: The above problem is an issue that occurs when you don’t have a trusted certificate when pulling images from Private registry.

Solution: We solve it with the -skip-verify parameter. Example command including it in the “k8s.io” namespace:

ctr --namespace k8s.io images pull xxx.harbor.com/apinizercloud/managerxxxx -skip-verify

Reason/Why: Kubernetes does not distribute pods in a balanced manner because by default, pods are placed on nodes that seem most suitable according to available resources, without a specific strategy or constraint.

Solution: Add the YAML file showing how pods will be distributed in a balanced manner using Pod Topology Spread Constraints after the second spec section.

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchExpressions:
          - key: node-role.kubernetes.io/control-plane
            operator: DoesNotExist

Check: You can check if the node-role.kubernetes.io/control-plane label exists on the node with the following command.

kubectl get nodes --show-labels

Reason/Why: When a node shuts down unexpectedly in Kubernetes (Non-Graceful Shutdown), Kubernetes Master detects this situation and performs necessary operations. However, this detection process may be delayed as it depends on the system’s timeout parameters.

Solution: The main parameters to consider for adjusting this time are:

1. Node Status Update Frequency

kubelet --node-status-update-frequency=5s

• The Node Status Update Frequency parameter determines how often Kubelet running on a node will update the node’s status to the Kubernetes API server, default value is 10s.
• We can enter this value lower than the default value for Kubelet to update node status more frequently, which allows Kubernetes to detect interruptions faster.

2. Node Monitor Grace Period

kube-apiserver --node-monitor-grace-period=20s

• The Node Monitor Grace Period parameter determines the maximum time the API server will wait before marking a node as “NotReady”, default value is 40s.
• This default value can be changed for the API server to mark the node as “NotReady” sooner or later.

3. Pod Eviction Timeout

kube-controller-manager --pod-eviction-timeout=2m

• Pod Eviction Timeout defines the maximum time to wait for pods on a node to be relocated (eviction) to other nodes after a node goes to “NotReady” state, default value is 5 minutes.
• The default value can be changed for pods on the node to be moved to other nodes faster.

Reason/Why: When a clone of a worker node already running in a Kubernetes cluster is added to the cluster, some configurations and credentials may conflict with the old node. These conflicts include:

• Duplicate machine-id values
• Old Kubernetes configuration remnants
• CNI and overlay network configuration conflicts

Solution:

1. Kubeadm reset. The cloned worker node’s existing cluster configuration is completely reset:

sudo kubeadm reset
sudo rm -rf $HOME/.kube

2. Kubernetes Created Overlay Network is Cleaned. CNI and other network components are cleaned:

rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1
systemctl restart containerd && systemctl restart kubelet

3. Cloned Machine’s machine-id is Reset. machine-id is regenerated according to the operating system:

# Command to change machine-id for RHEL
rm -f /etc/machine-id
systemd-machine-id-setup

# Command to change machine-id for Ubuntu
rm -f /etc/machine-id /var/lib/dbus/machine-id
systemd-machine-id-setup
cat /etc/machine-id > /var/lib/dbus/machine-id

4. Rejoining Cluster. Join command is obtained from master node and executed on cloned node:

kubeadm token create --print-join-command

Reason/Why: When the hostname of a worker node already running in a Kubernetes cluster is changed, some configurations and credentials may conflict with the old hostname. Therefore, when performing this operation, the worker node whose hostname will be changed should be removed from the cluster and added again after the hostname information is changed.

Solution:

1. Drain and Delete Node. Connect to master node in cluster:

kubectl get nodes
kubectl drain <NODES_OLD_HOSTNAME> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <NODES_OLD_HOSTNAME>

2. Connect to Worker Node Whose Hostname Will Change. After hostname is changed, CNI and other network components are cleaned:

sudo kubeadm reset
sudo hostnamectl set-hostname <NODES_NEW_HOSTNAME>
sudo reboot

hostname
# If the old hostname corresponding to 127.0.0.1 ip is still on /etc/hosts, this part should also be replaced with the new hostname
sudo vi /etc/hosts

rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/

systemctl stop kubelet && systemctl stop containerd

ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1

systemctl restart containerd && systemctl restart kubelet

3. Rejoining Cluster. Join command is obtained from master node and executed on cloned node:

kubeadm token create --print-join-command

Reason/Why: When Linux kernel detects an error in the underlying file system (ext4, xfs, etc.), it automatically puts the relevant disk partition into read-only mode to protect data integrity. When Kubernetes detects a disk error or other critical system issues on a node, it automatically adds a taint to prevent that node from accepting pods: node.kubernetes.io/unschedulable:NoSchedule

Solution:

1. Server is rebooted

sudo reboot

If the node has disk or system errors, some issues may be fixed after reboot.

2. Taint on node is removed

kubectl taint nodes <node-name> node.kubernetes.io/unschedulable:NoSchedule-

3. If node still doesn’t accept pods, uncordon is applied

kubectl uncordon <node-name>

The uncordon command makes the node schedulable and ensures that the unschedulable taint is removed in the background.

Reason/Why: This is an error encountered when Docker version is upgraded along with server package update. Kubernetes 1.18 does not support Docker version, causing incompatibility in communication between kubelet and Docker.

Solution: The problem is solved by downgrading Docker version to versions 18-19-20 compatible with Kubernetes 1.18.

# Check if image exists and inspect works
docker inspect k8s.gcr.io/kube-scheduler:v1.18.20

# Check Docker version
docker --version

# Remove existing docker packages
sudo apt remove -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# List suitable docker versions
apt-cache madison docker-ce | grep 19.03

# Install Docker 19.03 version
sudo apt install -y docker-ce=5:19.03.15~3-0~ubuntu-focal \
    docker-ce-cli=5:19.03.15~3-0~ubuntu-focal \
    containerd.io

# Check version
docker --version

# Packages are pinned with hold command to prevent incorrect updates
sudo apt-mark hold docker-ce docker-ce-cli containerd.io

# Docker and kubelet services are restarted
sudo systemctl restart docker.service
sudo systemctl restart kubelet.service

Reason/Why: Missing net.bridge.bridge-nf-call-iptables setting. Kubernetes’ iptables mode running kube-proxy component uses Linux bridges to route Service traffic. Having net.bridge.bridge-nf-call-iptables set to 0 prevents bridge traffic from being routed through iptables. This causes confusion in routing from Service ClusterIP to pod IPs. Kube-proxy usually gives the following warning in this case: “Missing br-netfilter module or unset sysctl br-nf-call-iptables…”

Solution: You need to enable Service routing by fixing the sysctl setting. Must Be Applied on All Kubernetes Nodes.

1) Adding Setting to Configuration File (or Checking): Make sure the following setting is set to 1. You can open the configuration file with the following command:

sudo vi /etc/sysctl.d/k8s.conf

Add the following line to the file (or edit it to 1 if it exists):

net.bridge.bridge-nf-call-iptables = 1

2) Loading Settings Immediately: Run the following command to load changes in the file to the system:

sudo sysctl --system

3) Verifying Settings: Use the following command to verify configuration:

sudo sysctl net.bridge.bridge-nf-call-iptables

Output should return 1.

4) Restarting Kube-proxy: Restart the pod with the following command so kube-proxy can receive new settings:

kubectl delete pod -n kube-system -l k8s-app=kube-proxy

When these steps are applied, timeout issues occurring in connections from Manager Pod to Worker Pod via ClusterIP will be resolved.