Learn Observability with the LGTM Stack - Deploying the LGTM Stack in Kubernetes

#Introduction

Building the LGTM Stack in Kubernetes differs from Docker Compose: every component becomes a separate workload, configuration becomes ConfigMaps or Secrets, and stateful data needs PersistentVolumes. Fortunately, Grafana provides official Helm charts for all components.

This episode covers the Helm chart for each component, the Kubernetes resources needed, high availability design, and operators and custom resources for declarative configuration.

#Helm Charts

#A Chart for Each Component

• Grafana Helm chart: grafana/grafana — deploys Grafana with provisioning.
• Loki distributed chart: grafana/loki in simple scalable mode.
• Tempo distributed chart: grafana/tempo supports single binary or microservices.
• Mimir distributed chart: grafana/mimir with separate components.
• Alloy Helm chart: grafana/alloy for a collector on every node.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

The helm repo add grafana command adds Grafana Labs' official chart repository.

#Deploying Components

#Mimir with Helm

An example Mimir deployment with the chart's default values:

helm install mimir grafana/mimir-distributed \
  --namespace mimir --create-namespace

After helm install mimir, the chart automatically creates StatefulSets for ingesters and Deployments for stateless components.

#The Kubernetes Resources Involved

• StatefulSets: for the stateful Mimir ingesters, Loki ingesters, and Tempo ingesters.
• Deployments: for the stateless distributor, querier, and compactor.
• Services: ClusterIP for internal communication, LoadBalancer for external access.
• Ingress: routes external traffic to Grafana.
• PersistentVolumeClaims: data storage for stateful components.
• ConfigMaps and Secrets: configuration and credentials for each component.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mimir-ingester-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

The PersistentVolumeClaim above provides 100Gi of storage for the Mimir ingester's data.

#High Availability Design

#Redundancy and Anti-Affinity

• Multi-replica deployments: at least 2-3 replicas for critical components.
• Pod anti-affinity: prevents replicas of stateful components from landing on the same node.
• Pod disruption budgets: keeps replicas available during node maintenance.
• Readiness and liveness probes: direct traffic only to healthy pods.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: mimir-ingester-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: mimir-ingester

minAvailable: 2 guarantees at least two ingesters keep running during maintenance.

#Operators and CRDs

#Prometheus Operator and ServiceMonitor

The Prometheus Operator introduces Custom Resource Definitions (CRDs) that make scraping configuration declarative:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout-monitor
spec:
  selector:
    matchLabels:
      app: checkout
  endpoints:
    - port: metrics

The ServiceMonitor tells the scraping system which pods to monitor — exactly the annotation concept from episode 27, but in resource form.

#Other Operators

• Grafana Operator: manages Grafana, data sources, and dashboards as CRDs.
• PodMonitor: like a ServiceMonitor for direct pod targets.

#Closing

In episode 28 you understood the Helm charts for each LGTM component, the Kubernetes resources involved from StatefulSet to PVC, high availability design with anti-affinity and PDBs, and operators and CRDs for declarative configuration.

The key takeaways:

• Use Grafana's official Helm charts for each component.
• Stateful ingesters need StatefulSets and PVCs.
• Anti-affinity and PDBs keep replicas available.
• ServiceMonitors and PodMonitors make scraping declarative.
• Start from chart default values, adjust gradually.

In the next episode 29 we'll discuss observability with a service mesh — automatic telemetry in Istio and Linkerd, Envoy metrics, automatic tracing, golden metrics, and mesh integration with the LGTM Stack. A service mesh brings observability with almost no manual instrumentation.