This document describes how to install Apinizer on Openshift 3.11 platform.

Please be sure to review the topology examples and review installing MongoDB and Elasticsearch applications separately from Openshift servers.

  • Replicaset MongoDB will be installed as version 4.2.0.
  • Elasticsearch will be installed as version 7.9.2.


#1) Operating System Configurations (All Servers)



# It is recommended that the following tools be installed on all servers.
sudo apt update
sudo apt install -y curl wget net-tools gnupg2 software-properties-common apt-transport-https ca-certificates

# The Apinizer user is created and authorized.
sudo adduser apinizer
sudo usermod -aG sudo apinizer

# Transactions are continued by switching to the user.
su - apinizer

# The firewall is turned off.
sudo systemctl stop ufw
sudo systemctl disable ufw

# Kubernetes, MongoDB and Elasticsearch jointly do not want the use of swap in the operating system. For that, let's disable swap.
# For operating system swap disabled operation.
sudo swapoff -a

# The swap line in the /etc/fstab file is deleted or commented so that incase of a reboot swap will not open.
# Then the file is closed (:wq)
sudo vi /etc/fstab
POWERSHELL

#2) Docker Installation


#2.1) Container Installation (Will be Done on All Openshift Servers)


Before proceeding to Kubernetes installation, the following steps are followed to prepare the system and install Docker.

#For the modules to be permanently installed 
sudo tee /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF

#For the modules to be installed on the running system 
sudo modprobe overlay
sudo modprobe br_netfilter
POWERSHELL

sysctl settings

sudo vi /etc/sysctl.d/k8s.conf
POWERSHELL

The first three lines here are mandatory, and the others can be changed according to the need.

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
net.ipv4.tcp_max_syn_backlog=40000
net.core.somaxconn=40000
net.core.wmem_default=8388608
net.core.rmem_default=8388608
net.ipv4.tcp_sack=1
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_moderate_rcvbuf=1
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_mem=134217728 134217728 134217728
net.ipv4.tcp_rmem=4096 277750 134217728
net.ipv4.tcp_wmem=4096 277750 134217728
net.core.netdev_max_backlog=300000
YML

Docker installation is done with the following codes.

sudo apt update

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update

sudo apt install -y containerd.io docker-ce docker-ce-cli

sudo mkdir -p /etc/systemd/system/docker.service.d

sudo tee /etc/docker/daemon.json <<EOF
{
  "insecure-registries" : [ "172.30.0.0/16" ],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
POWERSHELL

Docker service is started along with the final settings.

sudo systemctl daemon-reload 
sudo systemctl restart docker
sudo systemctl enable docker

sudo groupadd docker

sudo gpasswd -a $USER docker
POWERSHELL

#2.2) Kubernetes Installation (On Master)

wget https://github.com/openshift/origin/releases/download/v3.11.0/openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit.tar.gz
#Uncompress downloaded file.
tar xvf openshift-origin-client-tools*.tar.gz

cd openshift-origin-client*/
sudo mv  oc kubectl  /usr/local/bin/

#Verify installation of OpenShift client utility.
oc version

sudo systemctl restart docker

oc cluster up --public-hostname=YOURHOSTIP

oc login -u system:admin

oc adm policy add-cluster-role-to-user cluster-admin developer

oc login
POWERSHELL

#2.2.1) Bash Auto-Completion (Optional, On Any Openshift Master Server)


This process can speed up the writing of Openshift commands.

apt install bash-completion
source /usr/share/bash-completion/bash_completion
kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
POWERSHELL

#2.2.2) Setting User Configuration of Kubectl Command on Openshift Master Server (On Openshift Master Servers)


Definitions are made for the user who will run the kubectl commands

mkdir -p $HOME/.kube
sudo chown -R $(id -u):$(id -g) $HOME/.kube
POWERSHELL

#2.2.3) Install Kubernetes Network Plugin (On Openshift Master Servers)


In this guide, we will use the Flannel network add-on. You can choose other supported network add-ons. Flannel is a simple and easy way to configure a layer 3 network architecture for Kubernetes.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
POWERSHELL

Important

If you did not use the value 10.244.0.0/16 as podCIDR while initializing the Master, you should download the above yaml file and edit the network settings here as well.

#2.2.4) Installation Check (On Any Openshift Master Server )


If the Node created in addition to the Master can be seen when the following code is run on the Master, the installation has been completed successfully.

If it does not transition from NotReady to Ready status within two minutes, the problem should be investigated with the command 'kubectl describe node NODENAME'.

oc get node  

NAME         STATUS   ROLES    AGE   VERSION
localhost    Ready    <none>   5d    v1.11.0+d4cacc0     
BASH

#2.2.5) Defining Openshift Permissions (On Openshift Master Servers)


By default, Openshift deploys with at least one RBAC configuration to protect your cluster data. Currently, Dashboard only supports login with Bearer Token. Follow the steps below in order.

vi service.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
YML

vi adminuser.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
YML 
kubectl apply -f service.yaml

kubectl apply -f adminuser.yaml

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts

kubectl create clusterrolebinding apinizer -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:apinizer
POWERSHELL


#2.3) DNS Test (Optional, On Any Openshift Master Server)


https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#inheriting-dns-from-the-node

oc apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
POWERSHELL


#3) MongoDB Installation


#3.1) Operating System Configuration and Installation of MongoDB Application (On All MongoDB Servers)


wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list

sudo apt update
sudo apt install mongodb-org -y
POWERSHELL

#3.2) MongoDB Configurations (On All MongoDB Servers)


Edit the configuration file:

sudo vi /etc/mongod.conf
ip: 0.0.0.0
port: 25080

replication:
 replSetName: apinizer-replicaset

security:
    authorization: "enabled"

setParameter:
  transactionLifetimeLimitSeconds: 300
POWERSHELL

Then, the MongoDB application is started.

sudo systemctl start mongod
sudo systemctl enable mongod
POWERSHELL

#3.3) ReplicaSet Configuration and Authorization User Definition (MongoDB Primary Master Server)


Activating Replicaset

mongo mongodb://localhost:25080 
#At this stage, if it gives a connection error, server name with server address should be added to /etc/hosts and it should be checked whether one of the values of 127.0.0.1 is localhost
  
rs.initiate()
rs.status()
POWERSHELL

Creating an authorized user for Apinizer application.

use admin
db.createUser(
  {
    user: 'apinizer',
    pwd: '<YOUR_PASSWORD>',
    roles: [ { role: 'root', db: 'admin' } ],
	mechanisms:[ "SCRAM-SHA-1"] }
);

exit;
POWERSHELL

If you want to change the password

use admin

db.changeUserPassword("apinizer", passwordPrompt())
POWERSHELL 
 mongo  mongodb://localhost:25080 --authenticationDatabase "admin" -u "apinizer" -p

cfg = rs.conf()
cfg.members[0].host = "<MONGO_IP_ADDRESS>:25080"
rs.reconfig(cfg)
rs.status()
POWERSHELL

Authorize a user on the previously created MongoDB using the following command lines.

mongo mongodb://localhost:25080

use admin;
db.grantRolesToUser('admin', [{ role: 'root', db: 'admin' }])
POWERSHELL

#3.4) MongoDB ReplicaSet Installation on Multiple Servers (On MongoDB Slave Servers)


After the MongoDB installation, the keys folder created on the main node is moved to all nodes and the same permissions are given.

sudo openssl rand -base64 756 > /home/apinizer/mongo-key
sudo chmod 400 /home/apinizer/mongo-key
sudo chown -R mongodb:mongodb /home/apinizer/mongo-key
BASH

Copy the mongo-key file to all secondary nodes (mongoDb02, mongoDb03) in the location /home/apinizer/mongo-key

The mongod.conf file on all Mongo servers should be as follows.

On Node 1 => mongoDb01

# network interfaces
net:
	port: 25080
	bindIp: "127.0.0.1,mongoDb02,mongoDb03,k8sWorkerIP"
#security:
security:
	authorization: enabled
	keyFile:  /home/apinizer/mongo-key
#replication:
replication:
	replSetName: "apinizer-replicaset"
YML

After restarting the mongod services, the Secondary servers connect from Primary.

mongo  mongodb://localhost:25080 --authenticationDatabase "admin" -u "apinizer" -p 

rs.add("mongoDb02:25080")
rs.add("mongoDb03:25080")
BASH


#4) Elasticsearch Installation


#4.1) Operating System Configuration and Installation of Elasticsearch Application (On All Elasticsearch Servers)


sudo adduser elasticsearch
sudo usermod -aG sudo elasticsearch

 
sudo vi /etc/security/limits.conf
elasticsearch  -  nofile  65535
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
 
sudo sysctl -w vm.swappiness=1
sudo sysctl -w vm.max_map_count=262144
 
sudo vi /etc/sysctl.conf
vm.max_map_count=262144 elasticsearch
  
sudo sysctl -p
sudo sysctl vm.max_map_count
BASH

#4.2) Elasticsearch Installation (On All Elasticsearch Servers)


sudo mkdir /opt/elasticsearch
cd /opt/elasticsearch
sudo wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz
sudo tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
 
sudo chown -Rf elasticsearch:elasticsearch /opt/elasticsearch
sudo chmod -Rf 775 /opt/elasticsearch
 
##At this point, pay attention to where the appropriate disk is mounted or tell the system administrators to add the disk to the following path
#The commands below can be used for this purpose
df -h
lsblk
 
sudo mkdir /mnt/elastic-data/
sudo mkdir /mnt/elastic-snapdata/
 
sudo chown -Rf elasticsearch:elasticsearch /mnt/elastic-*
sudo chmod -Rf 775 /mnt/elastic-*
BASH

#4.3) Setting Elasticsearch Parameters According to the Environment (On All Elasticsearch Servers)


The following parameters must be adjusted and added according to your environment.

  • cluster.initial_master_nodes
  • network.host
  • node.name


sudo vi /opt/elasticsearch/elasticsearch-7.9.2/config/elasticsearch.yml
BASH

Important

Here, the path.data address should be given as the address of the disk in the system where your log file is added.

cluster.name: ApinizerEsCluster

node.name: "<ELASTICSEARCH_IP_ADDRESS>"
node.master: true
node.data: true

network.host: <ELASTICSEARCH_IP_ADDRESS>
http.port: 9200
 
cluster.initial_master_nodes: ["<ELASTICSEARCH_IP_ADDRESS>"]
 
discovery.seed_hosts: []
path.data: /mnt/elastic-data/
path.repo: ["/mnt/elastic-snapdata"]

bootstrap.memory_lock: true
 
http.cors.enabled : true
http.cors.allow-origin : "*"
http.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type, Content-Length
YML


You can set the JVM (Java Virtual Machine) values and other JVM parameters used by Elasticsearch as follows.

sudo vi /opt/elasticsearch/elasticsearch-7.9.2/config/jvm.options
BASH

Important

Here, it can be up to half of the amount of RAM the operating system has and this value should not exceed 32GB

-Xms8g
-Xmx8g
YML

#4.4) Setting Elasticsearch as Linux Service (On All Elasticsearch Servers)


sudo vi /opt/elasticsearch/elasticsearch-7.9.2/bin/elasticsearch-service.sh
BASH 
#!/bin/sh
SERVICE_NAME=elasticsearch
PATH_TO_APP="/opt/elasticsearch/elasticsearch-7.9.2/bin/$SERVICE_NAME"
PID_PATH_NAME="/opt/elasticsearch/elasticsearch-7.9.2/bin/$SERVICE_NAME.pid"
SCRIPTNAME=elasticsearch-service.sh
ES_USER=$SERVICE_NAME
ES_GROUP=$SERVICE_NAME
 
case $1 in
    start)
        echo "Starting $SERVICE_NAME ..."
        if [ ! -f $PID_PATH_NAME ]; then
        mkdir $(dirname $PID_PATH_NAME) > /dev/null 2>&1 || true
            chown $ES_USER $(dirname $PID_PATH_NAME)
            $SUDO $PATH_TO_APP -d -p $PID_PATH_NAME
        echo "Return code: $?"
            echo "$SERVICE_NAME started ..."
        else
            echo "$SERVICE_NAME is already running ..."
        fi
    ;;
    stop)
        if [ -f $PID_PATH_NAME ]; then
            PID=$(cat $PID_PATH_NAME);
            echo "$SERVICE_NAME stopping ..."
            kill -15 $PID;
            echo "$SERVICE_NAME stopped ..."
            rm $PID_PATH_NAME
        else
            echo "$SERVICE_NAME is not running ..."
        fi
    ;;
    restart)
        if [ -f $PID_PATH_NAME ]; then
            PID=$(cat $PID_PATH_NAME);
            echo "$SERVICE_NAME stopping ...";
            kill -15 $PID;
        sleep 1;
            echo "$SERVICE_NAME stopped ...";
            rm -rf $PID_PATH_NAME
            echo "$SERVICE_NAME starting ..."
            mkdir $(dirname $PID_PATH_NAME) > /dev/null 2>&1 || true
            chown $ES_USER $(dirname $PID_PATH_NAME)
            $SUDO $PATH_TO_APP -d -p $PID_PATH_NAME
            echo "$SERVICE_NAME started ..."
         else
            echo "$SERVICE_NAME is not running ..."
        fi
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|restart}" >&2
    exit 3
    ;;
esac
BASH 
sudo chmod -Rf 775 /opt/elasticsearch/elasticsearch-7.9.2/*

sudo vi /etc/systemd/system/elasticsearch.service
BASH 
[Unit]
Description=ElasticSearch Server
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target

[Service]
Type=forking
ExecStart=/opt/elasticsearch/elasticsearch-7.9.2/bin/elasticsearch-service.sh start
ExecStop=/opt/elasticsearch/elasticsearch-7.9.2/bin/elasticsearch-service.sh stop
ExecReload=/opt/elasticsearch/elasticsearch-7.9.2/bin/elasticsearch-service.sh restart
LimitNOFILE=65536
LimitMEMLOCK=infinity
User=elasticsearch
BASH 
sudo systemctl daemon-reload
sudo systemctl start elasticsearch
sudo systemctl status elasticsearch
sudo systemctl enable elasticsearch
BASH

You can use the following link for a compatible Kibana version

https://www.elastic.co/downloads/past-releases/kibana-oss-7-9-2
POWERSHELL

#5) Apinizer Installation

Refer to → Apinizer Installation and Configuration for Apinizer Installation.