Learn Observability with the LGTM Stack - Service Mesh Observability (Istio/Linkerd)

#Introduction

Adding instrumentation to every service is tedious work. A service mesh changes everything: a sidecar proxy accompanying every pod generates metrics, traces, and logs automatically without touching application code.

This episode covers automatic telemetry in Istio and Linkerd, Envoy metrics and golden metrics, automatic tracing, and how to integrate it all with the LGTM Stack.

#Service Mesh Telemetry

#Data Generated Automatically

• Automatic metrics generation: the proxy generates per-service and per-route metrics.
• Automatic distributed tracing: context is forwarded automatically between proxies.
• Access logs: logs every request passing through the proxy.
• Control plane metrics: the condition of the mesh itself, for example configuration and proxy health.

proxy -> metrics + trace + access log -> backend

The proxy -> metrics + trace + access log pattern is the core of the mesh advantage — three signals at once from a single layer.

#Observability in Istio

#Envoy Metrics and Telemetry v2

Istio collects metrics from the Envoy proxy:

• Envoy metrics: HTTP, TCP, and gRPC metrics per service.
• Istio telemetry v2: label-based metric aggregation like source, destination, and response code.
• Trace context propagation: W3C Trace Context forwarded automatically between proxies.
• Metrics aggregation: metrics from all proxies are collected for global queries.

istio_requests_total {
  source_service,
  destination_service,
  response_code
}

The istio_requests_total metric is the heart of Istio observability — its labels cover the entire request path between services.

#Grafana Dashboards for Istio

Grafana provides official Istio dashboards showing RPS, error rate, and latency per service — just connect them to the Mimir data source.

#Observability in Linkerd

#Golden Metrics and the Tap API

Linkerd focuses on simplicity with golden metrics:

• Success rate: the percentage of successful requests.
• Latency: the request duration distribution.
• RPS: requests per second.

Linkerd's special advantage is the Tap API, which inspects live traffic:

linkerd viz tap deploy/checkout

The linkerd viz tap deploy/checkout command displays every request passing through the checkout service in real time — very useful while debugging.

#Service Profiles and Tracing

• Service profiles: per-service definitions for latency and retries.
• Distributed tracing integration: forwards trace context to the backend — more limited than Istio, but enough for common needs.

#Integration with the LGTM Stack

#Connecting All the Data

To unify the mesh with LGTM:

• Scraping service mesh metrics: scrape the Istio or Linkerd metrics endpoint to Mimir.
• Trace backend configuration: point mesh traces to Tempo as the OTLP backend.
• Unified dashboards: dashboards combining mesh metrics and application metrics.
• Alerting on mesh metrics: alert rules monitoring mesh success rate and latency.

scrape_configs:
  - job_name: istio-metrics
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_istio]
        regex: ".*"
        action: keep

The job_name: istio-metrics configuration ensures mesh metrics enter Mimir along with other application metrics.

#Closing

In episode 29 you understood the automatic telemetry produced by a service mesh, Envoy metrics and istio_requests_total in Istio, golden metrics and the Tap API in Linkerd, and how to integrate the mesh with the LGTM Stack.

The key takeaways:

• Mesh proxies produce metrics, traces, and logs without instrumentation.
• Istio provides label-based metrics for source and destination.
• Linkerd offers golden metrics and the live Tap API.
• Scrape mesh metrics to Mimir and traces to Tempo.
• A mesh complements, not replaces, application instrumentation.

In the next episode 30 we'll discuss CI/CD and Infrastructure as Code — GitOps for dashboards and alert rules, the Grafana Terraform provider, automated testing in pipelines, and Jsonnet and Grafonnet for dashboards as code. Observability managed like code is the key to reproducibility.