Before the installation

Very Important

Before starting the installations, be sure to confirm from the system administrators that the servers are on the same network and on the same vm.

Very Important

Before starting the installations, make sure that the hostname of the server is not localhost.localdomain (with the hostname command). If this is the case, change it before starting the operations.

(If Required) Changing Hostname

hostnamectl set-hostname your-new-hostname

BASH

Very Important

There should be no "nameserver 127.0.1.1" entry in the /etc/resolv.conf file.

If Proxy is needed to access the internet, run the codes below on shell.

Very Important

Execute the followings on Linux Shell:

export http_proxy=http://proxyIp:port/ 
export https_proxy=http://proxyIp:port/
export no_proxy=localhost,127.0.0.1,SERVERIP,*.hostname

POWERSHELL

Add the below codes to correspondable files:

sudo vi /etc/apt/apt.conf

Acquire::http::Proxy "http://username:password@proxyIp:port";
Acquire::https::Proxy "https://username:password@proxyIp:port";

POWERSHELL

sudo vi /etc/systemd/system/docker.service.d/proxy.conf

[Service]
Environment="HTTP_PROXY=http://proxyIp:port"
Environment="HTTPS_PROXY=https://proxyIp:port"
Environment="NO_PROXY="localhost,127.0.0.1,::1,SERVERIP,*.hostname"

POWERSHELL

Very Important

On CentOS only, the following packages need to be deleted before starting the Docker installation.

yum remove podman* -y

yum remove buildah* -y

Important for Installation

In order for the installation to be healthy, Apinizer Kubernetes servers must be able to access the following addresses.

To access Docker Images:

https://hub.docker.com

To install Elasticsearch:

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz

To install Mongo:

http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/$basearch/

https://www.mongodb.org/static/pgp/server-4.2.asc

To install Kubernetes:

https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

https://packages.cloud.google.com/yum/doc/yum-key.gpg

https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

To install Kubernetes Dashboard:

https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml

SSL Inspection must be turned off on the firewall for the addresses below.

k8s.gcr.io
registry-1.docker.io
hub.docker.com

If the connection between servers will be restricted, the ports below must be add to exclude list:

6443/tcp # Kubernetes API server
2379-2380/tcp # etcd server client API
10250/tcp # Kubelet API
10251/tcp # kube-scheduler
10252/tcp # kube-controller-manager
8285/udp # Flannel
8472/udp # Flannel

30000-32767 #Applications on Kubernetes

Important

Note: While updating the packages, Ubuntu might try to pull from the server located in Turkey. However, from time to time, there may be a problem at tr.archive.ubuntu.com. In this case, you need to make the following change.

sudo vi /etc/apt/sources.list

//Replace all addresses with "tr."

Example:

Old: http://tr.archive.ubuntu.com/ubuntu

New: http://archive.ubuntu.com/ubuntu

Important

Steps 1, 2 and 3 need to be done on all servers that will run as Kubernetes Clusters.

1. Operating System Configurations

Ubuntu 2020.04

Centos 7.9

Necessary tools for development & diagnostic

sudo apt -y install curl apt-transport-https wget curl net-tools

POWERSHELL

Create Apinizer user

sudo adduser apinizer

sudo usermod -aG sudo apinizer

su - apinizer

POWERSHELL

Disable firewall

sudo systemctl stop ufw

sudo systemctl disable ufw

POWERSHELL

Deactivate Swap

Let's disable swap to avoid communication problems on nodes. For this, the following steps are done and the swap line in the /etc/fstab file is deleted.

sudo swapoff -a

sudo vi /etc/fstab

POWERSHELL

IP Tables Configurations

Then let's close the vi file (:wq). We will continue where we left off with the IPTables settings.

sudo vi /etc/sysctl.d/k8s.conf

POWERSHELL

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
net.ipv4.tcp_max_syn_backlog=40000
net.core.somaxconn=40000
net.core.wmem_default=8388608
net.core.rmem_default=8388608
net.ipv4.tcp_sack=1
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_moderate_rcvbuf=1
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_mem=134217728 134217728 134217728
net.ipv4.tcp_rmem=4096 277750 134217728
net.ipv4.tcp_wmem=4096 277750 134217728
net.core.netdev_max_backlog=300000

POWERSHELL

Save Changes

Let's apply the changes by running the following commands.

sudo modprobe br_netfilter

sudo sysctl --system

POWERSHELL

Loading br_netfilter module ( → Reboot)

Let's install the br_netfilter module and then restart it.

sudo lsmod | grep br_netfilter

sudo reboot

POWERSHELL

Necessary tools for development & diagnostic

sudo yum install -y net-tools yum-utils bind-utils  device-mapper-persistent-data lvm2 telnet wget zip curl lsof

POWERSHELL

Create Apinizer user

adduser apinizer  

sudo usermod -aG wheel apinizer

sudo passwd apinizer

su - apinizer

POWERSHELL

Disable firewall

sudo systemctl stop firewalld

sudo systemctl disable firewalld

POWERSHELL

Deactivate Selinux

Let's disable SELinux to avoid communication problems on servers.

sudo setenforce 0

sudo sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

POWERSHELL

Deactivate Swap

Let's disable swap to avoid communication problems on nodes. For this, the following steps are done and the swap line in the /etc/fstab file is deleted.

sudo swapoff -a

sudo vi /etc/fstab

POWERSHELL

IP Tables Configurations

Then let's close the vi file (:wq). We will continue where we left off with the IPTables settings.

sudo vi /etc/sysctl.d/k8s.conf

POWERSHELL

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
net.ipv4.tcp_max_syn_backlog=40000
net.core.somaxconn=40000
net.core.wmem_default=8388608
net.core.rmem_default=8388608
net.ipv4.tcp_sack=1
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_moderate_rcvbuf=1
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_mem=134217728 134217728 134217728
net.ipv4.tcp_rmem=4096 277750 134217728
net.ipv4.tcp_wmem=4096 277750 134217728
net.core.netdev_max_backlog=300000

POWERSHELL

Save Changes

Let's apply the changes by running the following commands.

sudo modprobe br_netfilter

sudo sysctl --system

POWERSHELL

Loading br_netfilter module ( → Reboot)

Let's install the br_netfilter module and then restart it.

sudo lsmod | grep br_netfilter

sudo reboot

POWERSHELL

2. Docker Installation

Run the following command to install Docker before proceeding with the Kubernetes installation.

Ubuntu 2020.04

Centos 7.9

sudo apt update

sudo apt install -y gnupg2 software-properties-common apt-transport-https ca-certificates

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update

sudo apt install -y containerd.io docker-ce docker-ce-cli

sudo mkdir -p /etc/systemd/system/docker.service.d

sudo tee /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

POWERSHELL

Start the docker

sudo systemctl daemon-reload 
sudo systemctl restart docker
sudo systemctl enable docker

sudo groupadd docker

sudo gpasswd -a $USER docker

POWERSHELL

sudo yum-config-manager \
--add-repo https://download.docker.com/linux/centos/docker-ce.repo
 
sudo yum install docker-ce --nobest -y
 
sudo mkdir /etc/docker
 
sudo vi /etc/docker/daemon.json 
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}

sudo mkdir -p /etc/systemd/system/docker.service.d
 
sudo systemctl daemon-reload
sudo systemctl start docker
sudo systemctl enable docker

sudo usermod -aG docker apinizer

POWERSHELL

3. Kubernetes Installation

Ubuntu 2020.04

Centos 7.9

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update

POWERSHELL

sudo apt -y install kubelet=1.18.4-00 kubeadm=1.18.4-00 kubectl=1.18.4-00
sudo apt-mark hold kubelet kubeadm kubectl

POWERSHELL

sudo systemctl enable kubelet

sudo systemctl start kubelet

POWERSHELL

sudo vi /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

POWERSHELL

sudo yum install -y kubelet-1.18.4 kubeadm-1.18.4 kubectl-1.18.4

POWERSHELL

sudo systemctl enable kubelet

sudo systemctl start kubelet

POWERSHELL

Beware

All of the above steps should be done for all servers in Kubernetes Cluster.

4. Setup Kubernetes Master Node(s)

on Master Node

Beware

The following steps should only be done on the Kubernetes Master Server.

Setting Single-Master Kubernetes (Skip the next instruction!)

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=<MASTERSERVERIPADDRESS>

POWERSHELL

Setting Multi-Master Kubernetes (Previous instruction must be skipped!)

For the Load Balance address here, the servers to be defined as the Master must be built behind a Load Balancer.

sudo kubeadm init --kubernetes-version "1.18.4" --pod-network-cidr "10.244.0.0/16" --control-plane-endpoint "<MASTERSERVERIPADDRESS>:6443" --upload-certs

POWERSHELL

For using Multi-Master structure, join the other Master nodes with code below

sudo kubeadm join <MASTERSERVERIPADDRESS>:6443 --token xxx --discovery-token-ca-cert-hash sha256:yyy --control-plane --certificate-key zzz

BASH

Very Important

#If it needs to recreated, take the results of the code below and add the second to first;

kubeadm token create --print-join-command

sudo kubeadm init phase upload-certs --upload-certs

#It will look like this:

<join command from step 1> --control-plane --certificate-key <key from step 2>

#If you want to manually build the phrase, use the followings:

for xxx → kubeadm token list

for yyy → openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

for zzz → sudo kubeadm init phase upload-certs --upload-certs

Changing the kubectl-authorized user to apinizer user.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown -R $(id -u):$(id -g) $HOME/.kube

POWERSHELL

Set up Pod Network

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

POWERSHELL

Making Master also work as Worker on single server installations;

Using Master node also as Worker node

Kubernetes single control-plane configuration is done.

When the following piece of code is run, the Master node can also be run as a Worker node on a single machine. It is not a recommended installation method. There must be at least 2 servers.

kubectl taint nodes --all node-role.kubernetes.io/master-

POWERSHELL

To remove the master without deleting the node

kubectl taint nodes node-name dedicated:NoSchedule-

POWERSHELL

5. Registering Kubernetes Worker Nodes to Kubernetes Master Node

A token information is needed to connect the Worker server to the Master. This will be seen as written on the master node during the installation phase. But if it is omitted or you want to view it again, the following command can be used.

Run On Master Node

sudo kubeadm token create --print-join-command

POWERSHELL

Affiliate the Worker node/s to Master node

sudo kubeadm join <MASTERSERVERIPADDRESS>:6443 --token xxx --discovery-token-ca-cert-hash sha256:yyy

BASH

6. Checking the Result

When the code below is run from the Master, if the Node created in addition to the Master is also visible, the installation has been completed successfully.

If it does not switch from NotReady to Ready after two minutes, the problem should be investigated with the "kubectl describe node NODENAME" command.

kubectl get node

NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   5d    v1.18.4
ks-worker1   Ready    <none>   5d    v1.18.4

BASH

7. Registering Kubernetes Authorizations

By default, Kubernetes deploys with at least one RBAC configuration to protect your cluster data. Currently, Dashboard only supports login with Bearer Token. Follow the steps below in order.

on Master Node

vi service.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system

YML

on Master Node

vi adminuser.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

YML

on Master Node

kubectl apply -f service.yaml

kubectl apply -f adminuser.yaml

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts

kubectl create clusterrolebinding apinizer -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:apinizer

POWERSHELL

DNS Test (Optional)

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#inheriting-dns-from-the-node

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

kubectl get pods --namespace=kube-system -l k8s-app=kube-dns

kubectl -n kube-system get configmap coredns -oyaml

kubectl exec -i -t dnsutils -- nslookup kubernetes.default

kubectl exec -ti dnsutils -- cat /etc/resolv.conf

POWERSHELL