Installing Docker and Kubernetes
Before the installation
Very Important
Before starting the installations, be sure to confirm from the system administrators that the servers are on the same network and on the same vm.
Very Important
Before starting the installations, make sure that the hostname of the server is not localhost.localdomain (with the hostname command). If this is the case, change it before starting the operations.
(If Required) Changing Hostname
hostnamectl set-hostname your-new-hostname
Very Important
There should be no "nameserver 127.0.1.1" entry in the /etc/resolv.conf file.
If Proxy is needed to access the internet, run the codes below on shell.
Very Important
Execute the followings on Linux Shell:
export http_proxy=http://proxyIp:port/
export https_proxy=http://proxyIp:port/
export no_proxy=localhost,127.0.0.1,SERVERIP,*.hostname
Add the below codes to correspondable files:
sudo vi /etc/apt/apt.conf
Acquire::http::Proxy "http://username:password@proxyIp:port";
Acquire::https::Proxy "https://username:password@proxyIp:port";
sudo vi /etc/systemd/system/docker.service.d/proxy.conf
[Service]
Environment="HTTP_PROXY=http://proxyIp:port"
Environment="HTTPS_PROXY=https://proxyIp:port"
Environment="NO_PROXY="localhost,127.0.0.1,::1,SERVERIP,*.hostname"
Very Important
On CentOS only, the following packages need to be deleted before starting the Docker installation.
yum remove podman* -y
yum remove buildah* -y
Important for Installation
In order for the installation to be healthy, Apinizer Kubernetes servers must be able to access the following addresses.
To access Docker Images:
To install Elasticsearch:
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz
To install Mongo:
http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/$basearch/
https://www.mongodb.org/static/pgp/server-4.2.asc
To install Kubernetes:
https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
To install Kubernetes Dashboard:
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
SSL Inspection must be turned off on the firewall for the addresses below.
k8s.gcr.io
registry-1.docker.io
hub.docker.com
If the connection between servers will be restricted, the ports below must be add to exclude list:
6443/tcp # Kubernetes API server
2379-2380/tcp # etcd server client API
10250/tcp # Kubelet API
10251/tcp # kube-scheduler
10252/tcp # kube-controller-manager
8285/udp # Flannel
8472/udp # Flannel
30000-32767 #Applications on Kubernetes
Important
Note: While updating the packages, Ubuntu might try to pull from the server located in Turkey. However, from time to time, there may be a problem at tr.archive.ubuntu.com. In this case, you need to make the following change.
sudo vi /etc/apt/sources.list
//Replace all addresses with "tr."
Example:
Important
Steps 1, 2 and 3 need to be done on all servers that will run as Kubernetes Clusters.
1. Operating System Configurations
Ubuntu 2020.04 | Centos 7.9 |
---|---|
Necessary tools for development & diagnostic
POWERSHELL
Create Apinizer user
POWERSHELL
Disable firewall
POWERSHELL
Deactivate SwapLet's disable swap to avoid communication problems on nodes. For this, the following steps are done and the swap line in the /etc/fstab file is deleted.
POWERSHELL
IP Tables ConfigurationsThen let's close the vi file (:wq). We will continue where we left off with the IPTables settings.
POWERSHELL
POWERSHELL
Save ChangesLet's apply the changes by running the following commands.
POWERSHELL
Loading br_netfilter module ( → Reboot)Let's install the br_netfilter module and then restart it.
POWERSHELL
| Necessary tools for development & diagnostic
POWERSHELL
Create Apinizer user
POWERSHELL
Disable firewall
POWERSHELL
Deactivate SelinuxLet's disable SELinux to avoid communication problems on servers.
POWERSHELL
Deactivate SwapLet's disable swap to avoid communication problems on nodes. For this, the following steps are done and the swap line in the /etc/fstab file is deleted.
POWERSHELL
IP Tables ConfigurationsThen let's close the vi file (:wq). We will continue where we left off with the IPTables settings.
POWERSHELL
POWERSHELL
Save ChangesLet's apply the changes by running the following commands.
POWERSHELL
Loading br_netfilter module ( → Reboot)Let's install the br_netfilter module and then restart it.
POWERSHELL
|
2. Docker Installation
Run the following command to install Docker before proceeding with the Kubernetes installation.
Ubuntu 2020.04 | Centos 7.9 |
---|---|
POWERSHELL
Start the docker
POWERSHELL
|
POWERSHELL
|
3. Kubernetes Installation
Ubuntu 2020.04 | Centos 7.9 |
---|---|
POWERSHELL
POWERSHELL
POWERSHELL
|
POWERSHELL
POWERSHELL
POWERSHELL
|
Beware
All of the above steps should be done for all servers in Kubernetes Cluster.
4. Setup Kubernetes Master Node(s)
on Master Node
Beware
The following steps should only be done on the Kubernetes Master Server.
Setting Single-Master Kubernetes (Skip the next instruction!)
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=<MASTERSERVERIPADDRESS>
Setting Multi-Master Kubernetes (Previous instruction must be skipped!)
For the Load Balance address here, the servers to be defined as the Master must be built behind a Load Balancer.
sudo kubeadm init --kubernetes-version "1.18.4" --pod-network-cidr "10.244.0.0/16" --control-plane-endpoint "<MASTERSERVERIPADDRESS>:6443" --upload-certs
For using Multi-Master structure, join the other Master nodes with code below
sudo kubeadm join <MASTERSERVERIPADDRESS>:6443 --token xxx --discovery-token-ca-cert-hash sha256:yyy --control-plane --certificate-key zzz
Very Important
#If it needs to recreated, take the results of the code below and add the second to first;
kubeadm token create --print-join-command
sudo kubeadm init phase upload-certs --upload-certs
#It will look like this:
<join command from step 1> --control-plane --certificate-key <key from step 2>
#If you want to manually build the phrase, use the followings:
for xxx → kubeadm token list
for yyy → openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
for zzz → sudo kubeadm init phase upload-certs --upload-certs
Changing the kubectl-authorized user to apinizer user.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown -R $(id -u):$(id -g) $HOME/.kube
Set up Pod Network
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Making Master also work as Worker on single server installations;
Using Master node also as Worker node
Kubernetes single control-plane configuration is done.
When the following piece of code is run, the Master node can also be run as a Worker node on a single machine. It is not a recommended installation method. There must be at least 2 servers.
kubectl taint nodes --all node-role.kubernetes.io/master-
To remove the master without deleting the node
kubectl taint nodes node-name dedicated:NoSchedule-
5. Registering Kubernetes Worker Nodes to Kubernetes Master Node
A token information is needed to connect the Worker server to the Master. This will be seen as written on the master node during the installation phase. But if it is omitted or you want to view it again, the following command can be used.
Run On Master Node
sudo kubeadm token create --print-join-command
Affiliate the Worker node/s to Master node
sudo kubeadm join <MASTERSERVERIPADDRESS>:6443 --token xxx --discovery-token-ca-cert-hash sha256:yyy
6. Checking the Result
When the code below is run from the Master, if the Node created in addition to the Master is also visible, the installation has been completed successfully.
If it does not switch from NotReady to Ready after two minutes, the problem should be investigated with the "kubectl describe node NODENAME" command.
kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 5d v1.18.4
ks-worker1 Ready <none> 5d v1.18.4
7. Registering Kubernetes Authorizations
By default, Kubernetes deploys with at least one RBAC configuration to protect your cluster data. Currently, Dashboard only supports login with Bearer Token. Follow the steps below in order.
on Master Node
vi service.yaml
apiVersion: v1
kind: Namespace
metadata:
name: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
on Master Node
vi adminuser.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
on Master Node
kubectl apply -f service.yaml
kubectl apply -f adminuser.yaml
kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts
kubectl create clusterrolebinding apinizer -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:apinizer
DNS Test (Optional)
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
kubectl -n kube-system get configmap coredns -oyaml
kubectl exec -i -t dnsutils -- nslookup kubernetes.default
kubectl exec -ti dnsutils -- cat /etc/resolv.conf