1-) Grafana Setup

Persistent Storage Configuration

Since Grafana's data will be stored on a node in a Kubernetes cluster, PersistentVolume (PV) and PersistentVolumeClaim (PVC) definitions are required. This configuration allows Grafana to protect its data in case of shutdown or restart.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
  namespace: monitoring
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/data/grafana
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  namespace: monitoring
spec
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

YML

The PersistentVolume (PV) and PersistentVolumeClaim (PVC) configuration above allows Grafana to store its data on a specific node. However, the hostPath used here is dependent on the local file system of the specific node where Grafana is running.

Therefore:

If Grafana pods are moved to a different node, they will lose their data unless they have the same hostPath directory on the new node.
To guarantee that pods always run on the same node, it is necessary to pin pods to specific nodes using nodeAffinity or nodeSelector.

Alternatively, NFS, Ceph, Longhorn or a cloud-based storage solution can be used to store data in a node-independent manner.

Grafana ConfigMap Configuration

The username and password to be used in the Grafana interface are stored in the following configmap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-config
  namespace: monitoring
data:
  GF_SECURITY_ADMIN_USER: admin
  GF_SECURITY_ADMIN_PASSWORD: admin123

YML

You can also keep the login user for the graphana interface as Secret instead of ConfigMap.

Modify the Grafana Deployment YAML file below to suit your systems and install it on your Kubernetes Cluster.

Grafana Deployment Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      initContainers:
        - name: init-permissions
          image: busybox
          command: ["sh", "-c", "chown -R 472:472 /var/lib/grafana"]
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-storage
      containers:
        - name: grafana
          image: grafana/grafana:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /api/health
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 5
          livenessProbe:
            httpGet:
              path: /api/health
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          envFrom:
            - configMapRef:
                name: grafana-config
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-storage
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
            claimName: grafana-pvc

YML

Kubernetes Service is created for Grafana.

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
  labels:
    app: grafana
spec:
  ports:
    - name: http
      port: 3000
      nodePort: 32181
  selector:
    app: grafana
  type: NodePort

YML

When Grafana is deployed on Kubernetes, it creates a Kubernetes service named grafana and of type NodePort. This service is required for access to Grafana from outside Kubernetes. However, instead of this service, you can use Ingress or whatever structure you use for the connection method in your organization.

2) Setting Prometheus as Data Source on Grafana

1. Sign in to Grafana

2. Click on the Data Source tab in the Left Menu and select Add new data source.

3. Select Prometheus as data source.

4. Enter and save the Prometheus Connection url and other required settings.

5. Click on the Dashboard tab from the Left Menu and select new dashboard.

6. Create a new dashboard and select prometheus as data source.