Learn Observability with the LGTM Stack - Kubernetes Observability

#Introduction

Pods that come and go within seconds, IPs that always change, and nodes scaling up and down — Kubernetes is the most dynamic environment for observability. Static monitoring approaches simply don't apply here.

This episode covers Kubernetes monitoring architecture, metric sources, service discovery with relabeling, pod log collection, and cross-service tracing within the cluster.

#Kubernetes Monitoring Architecture

#Four Monitoring Layers

• Node-level monitoring: system metrics from kubelet and node exporter.
• Pod-level monitoring: metrics from each pod and container.
• Container metrics: CPU, memory, and I/O per container.
• Cluster-level monitoring: the condition of the whole cluster, including the scheduler and API server.

node -> pod -> container -> cluster

The node -> pod -> container -> cluster pattern is the thinking map for designing K8s observability dashboards.

#Kubernetes Metric Sources

#Where Metrics Come From

• cAdvisor metrics: built-in container metrics from the kubelet.
• Kubelet metrics: node health and operation metrics.
• kube-state-metrics: the status of K8s objects like deployments, pods, and HPAs.
• Metrics Server API: short metrics for the HPA.
• Control plane metrics: API server, scheduler, and controller-manager metrics.

curl -sk https://<node>:10250/metrics

The curl -sk .../metrics command shows raw kubelet metrics — in production, scraping is done automatically by service discovery.

#Service Discovery and Relabeling

#Kubernetes SD

Prometheus and Mimir use Kubernetes service discovery to find targets automatically:

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        regex: "true"
        action: keep

The role: pod configuration makes the scraper find all pods, then relabeling filters pods that have the scrape annotation enabled.

#Annotations and ServiceMonitors

• Pod annotations: prometheus.io/scrape: "true" marks pods that must be scraped.
• Service monitors: Prometheus Operator CRDs that define targets declaratively — covered in episode 28.

#Pod Log Collection

#DaemonSet and Sidecar

Pod logs are collected with two main patterns:

• DaemonSet pattern: one collector per node (for example Grafana Alloy) reads the logs of all pods on that node.
• Sidecar pattern: one collector container bundled with the application pod.

kind: DaemonSet
metadata:
  name: alloy
spec:
  template:
    spec:
      containers:
        - name: alloy
          image: grafana/alloy:latest

kind: DaemonSet ensures one Alloy instance runs on every node of the cluster.

#Events and Audit Logs

• Event logs: K8s events like pod restarts and scheduling.
• Audit logs: records of requests to the API server — important for compliance in episode 33.

#Tracing in Kubernetes

#Service Mesh and Context Propagation

Tracing across pods requires automatic context propagation:

• Service mesh integration: Istio and Linkerd inject tracing automatically — covered in episode 29.
• Ingress controller tracing: traces start at the cluster entry point.
• Pod instrumentation: the OTel SDK inside pods forwards context via headers.

ingress -> pod A -> pod B -> pod C -> backend

The ingress -> pod A -> pod B -> pod C pattern is a single request's trace within a cluster — without propagation, this chain breaks at every hop.

#Closing

In episode 27 you understood the architecture for monitoring nodes, pods, containers, and clusters, the metric sources cAdvisor, kubelet, and kube-state-metrics, service discovery and relabeling with annotations, pod log collection with DaemonSet and sidecar, and cross-pod tracing.

The key takeaways:

• Kubernetes demands discovery-based monitoring, not static.
• cAdvisor, kubelet, and kube-state-metrics are the main metric sources.
• Annotations and ServiceMonitors mark scrape targets.
• The Alloy DaemonSet collects logs per node.
• A service mesh automates cross-pod tracing.

In the next episode 28 we'll discuss deploying the LGTM Stack in Kubernetes — Helm charts for each component, K8s resources like StatefulSet and PersistentVolumeClaim, high availability design, and using operators. Your local stack will move into a real cluster.