Possible problems that may occur on Apinizer.




Type

Useful commands

Linux

//sistem saatini "date" komutuyla kontrol ederek yanlış bölgede olması durumunda:

sudo ln -sf /usr/share/zoneinfo/Europe/Istanbul /etc/localtime

//Doğru bölgede ancak hala yanlış olması durumunda:

date -s '2014-12-25 12:34:56'

Kubernetes

journalctl -xeu kubelet

kubectl get events --all-namespaces --sort-by='.metadata.creationTimestamp'

kubectl describe pod -n PODNAMESPACE PODNAME

kubectl exec -it podName -n prod -- curl x.y.gov.tr

Mongo

db.users.find( { name: "Joe" } );

db.db_to_api.find().pretty();

mongodump --authenticationDatabase "admin" --port 25080 -d apinizerdb -u apinizer -p PASSWORD

sudo tail -999f /var/log/mongodb/mongod.log

Elasticsearchcurl -XPUT '10.1.1.1:9200/*log*/_settings' -H 'Content-Type: application/json' -d'{"index": {"blocks": {"read_only_allow_delete": null}}}'
curl

curl -D -  --header "Content-Type:text/xml;charset=UTF-8" --header "SOAPAction:XXX" -d '#SOAP BODY GOES HERE#' https://api.address/apigateway/relativepath >> response.txt

//Skip the ssl: -k

//logging,verbose: -v

//timeout: -m 10

//Sending username and let it ask for password: -u abc

//Saving the response: --output response.txt


Problem

MongoDb installation fails on Centos 8

Cause/Explanation

If you have any output like the following in “systemctl status mongodb.service -l”:

SELinux is preventing /usr/bin/mongod from read access on the file snmp.
Solution

run these commands and see output of mongodb.service status until errors disappear:

grep mongod /var/log/audit/audit.log | audit2allow -M mypol
semodule -i mypol.pp
grep ftdc /var/log/audit/audit.log | audit2allow -M mypol
semodule -i mypol.pp
Problem

"Coğrafi Bilgi Sistemleri" services needs to change some addresses on response.

Cause/Explanation-
Solution

A Business Rule or Script policy needs to be used to change the constant url's inside the response so it will contains the apinizer address.

Problem

On traffic logs, Client IP addresses seen as 10.244.x.x

Cause/ExplanationKubernetes NodePort yapısında eğer client'dan X-Forwarded-For değer gelmiyorsa default olarak arka taraftaki uygulamaya pod'un IP adresini yönlendiriyor. Çözüm olarak externalTrafficPolicy değer Local yaparak çözebiliyoruz ancak bu defa NodePort gelen isteği Cluster'daki diğer sunuculara yönlendirmediğinden erişim sıkıntısı oluşuyor. İlgili Node'u bilip ona gitmek gerekiyor.
Solution

Apinizer Worker'ları Nginx veya F5 gibi bir yapının arkasına konacağı için, ilgili loadbalancer'ın konfigürasyon dosyasına aşağıdaki gibi header'a xff bilgisinin eklenmesi gerekiyor.


location /apigateway/ {proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://YOUR-IP:PORT/apigateway/;
} # end location
Problem

Error on Installing docker to Centos 8.3.x  servers

Cause/Explanation

RHEL 8 ve CentOS 8 uses default distributions of podman and buildah. These need to be removed but note that Redhat decided to not support Docker officially.

Solution$ yum remove podman* -y 

$ yum remove buildah* -y
Problem

docker: Error response from daemon: Get https://registry-1.docker.io/v2/: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kurumSertifikasıAdı-CA").

Cause/ExplanationFirewall adding its own certificate doing ssl inspection.
Solutiondocker.io'u firewall üzerinde "ssl inspection exception"'a eklenecek.
Problem

Node stucks on NotReady ve "Unable to update cni config: no networks found in /etc/cni/net.d"

Cause/Explanation

kube-flannel can not create necessary files and folders on Master.

Solution

(Alternative solution/s: https://github.com/kubernetes/kubernetes/issues/54918)

$ sudo mkdir -p /etc/cni/net.d

$ sudo vi /etc/cni/net.d/10-flannel.conflist

#aşağıdaki eklenir.

{

"name": "cbr0",

"plugins": [

{

"type": "flannel",

"delegate": {

"hairpinMode": true,

"isDefaultGateway": true

}

},

{

"type": "portmap",

"capabilities": {

"portMappings": true

}

}

]

}

----------

{

"name": "cbr0",

"cniVersion": "0.3.1",

"plugins": [

{

"type": "flannel",

"delegate": {

"isDefaultGateway": true

}

},

{

"type": "portmap",

"capabilities": {

"portMappings": true

}

}

]

}

------------

sudo chmod -Rf 777 /etc/cni /etc/cni/*

sudo chown -Rf apinizer:apinizer /etc/cni /etc/cni/*


sudo systemctl daemon-reload

sudo systemctl restart kubelet


#To check are there pods that still can not pull some images

kubectl get pods -n kube-system

describe pod podAdi -n kube-system

Problem

kubeadm error – kubelet isn’t running or healthy and connection refused

Cause/Explanation-
Solution

sudo swapoff -a sudo sed -i '/ swap / s/^/#/' /etc/fstab

sudo reboot

kubeadm reset kubeadm init --ignore-preflight-errors all

Problem

Docker pull gives x509 certificate error

Cause/Explanation

If the company does not use https, do the below on all Nodes.

Solution

$ sudo vi /etc/docker/daemon.json

"insecure-registries" : ["hub.docker.com:443", "registry-1.docker.io:443", "quay.io"]

sudo systemctl daemon-reload sudo systemctl restart docker

#Aşağıdaki ile kontrol edilir.
docker info
Cause/Explanation

If the company use https, add their ssl certifitace ("crt") to all Nodes.

Solution

cp ssl.crt /usr/local/share/ca-certificates/
update-ca-certificates
service docker restart


#Centos 7
sudo cp -p ssl.crt /etc/pki/ca-trust/source
sudo cp ssl.crt /etc/pki/ca-trust/source/anchors/myregistrydomain.com.crt

sudo update-ca-trust extract
sudo systemctl daemon-reload
sudo systemctl restart docker

Problem

Nexus proxy

Cause/ExplanationIf the company uses Nexus Proxy, below is needed to be done on all Nodes.
Solution

$ sudo vi /etc/docker/daemon.json

{

"data-root":"/docker-data",

"insecure-registries":["nexusdocker.kurumunadresi.com.tr"],

"registry-mirrors":["https://nexusdocker.kurumunadresi.com.tr"],

"exec-opts": ["native.cgroupdriver=systemd"],

"log-driver": "json-file",

"log-opts": {

"max-size": "100m"

},

"storage-driver": "overlay2"

}

Problem

Kubernetes DNS Problem (connection timed out; no servers could be reached)

Cause/ExplanationNode stays on Ready,SchedulingDisabled
Solution
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

kubectl get pods dnsutils

kubectl exec -i -t dnsutils -- nslookup kubernetes.default

//The response below means it is all correct

Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      kubernetes.default
Address 1: 10.0.0.1

//The response below means there are some configuration errors
Server: 10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1


//Check the Resolv.conf

kubectl exec -ti dnsutils -- cat /etc/resolv.conf

//Correct

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local kurum.gov.tr
options ndots:5

//Incorrect

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

kubectl rollout restart -n kube-system deployment/coredns

Solution 2

It may also be resolved by adding the domain name to /etc/resolv.conf

search company.com.tr

Problem

MongoDb installation error on Centos 7

Cause/Explanation

warning: /var/cache/yum/x86_64/7/MongoDB/packages/mongodb-org-mongos-4.2.13-1.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 058f8b6b: NOKEY
Retrieving key from https://www.mongodb.org/static/pgp/server-4.2.asc

Solution

//Doing gpgcheck=0 on file below

sudo vi /etc/yum.repos.d/mongodb.repo


[MongoDB]
name=MongoDB Repository
baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/$basearch/
gpgcheck=0
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc


Problem

Client certificates generated by kubeadm expires after 1 year

Cause/Explanation

Unable to connect to the server: x509: certificate has expired or is not yet valid

Solution
#these have to be done on all masters

sudo kubeadm alpha certs check-expiration
sudo kubeadm alpha certs renew all

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

#further readings:
https://serverfault.com/questions/1065444/how-can-i-find-which-kubernetes-certificate-has-expired)
https://www.oak-tree.tech/blog/k8s-cert-yearly-renewwal