Learn Istio - Observability at Scale & Correlation

#Introduction

At small scale, Prometheus can store all metrics and everything looks clear. As the mesh grows, telemetry costs explode: high-cardinality metrics eat storage, traces flood the backend, and alerts risk becoming noise. Episode 20 covers how to manage observability that survives at large scale and how to build SLOs that are actually measured.

#Handling High-Cardinality Metrics

#Sources of Cardinality

The istio_requests_total metric has many labels: destination_service, response_code, response_flags, source_workload, and more. With thousands of services, the label value combinations become millions of series. Uncontrolled cardinality is Prometheus's number one enemy.

#Shrinking the Dimensions

Do not remove labels blindly. Cut the ones not used for decisions:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: metric-reduce
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    - match:
        metric: REQUEST_COUNT
      tagOverrides:
        connection_security_policy:
          operation: REMOVE

tagOverrides with operation: REMOVE removes the connection_security_policy dimension from REQUEST_COUNT. The principle: keep the dimensions used for SLIs and debugging, discard the rest. Measure the effect on the series count in Prometheus.

#Aggregation and Downsampling

Use Prometheus recording rules to compute aggregate metrics (for example the error rate per service per hour) and keep raw metrics for a shorter period. This trims heavy queries and long-term storage.

#Trace Sampling Strategies

In episode 8 we used 10 percent random sampling. At large scale, sampling needs to be smarter:

• Head-based sampling: the decision happens at the start of the trace (at the client), simple but can miss important traces.
• Tail-based sampling: the decision happens after the trace finishes, keeping traces with errors or extreme latency — more accurate, more expensive.
• Prioritized sampling: always sample traces containing errors, critical endpoints, or important accounts.

spec:
  tracing:
  - providers:
    - name: otel
    randomSamplingPercentage: 5
    useRequestIdForTraceSampling: true

useRequestIdForTraceSampling: true makes the sampling decision consistent for a single request across all hops. The goal: enough trace data for debugging without flooding storage.

#Log Aggregation

Envoy access logs are numerous. A healthy aggregation strategy:

• Switch the log format to JSON for easy parsing.
• Send logs to a centralized pipeline (for example Loki or Elasticsearch).
• Apply log sampling to 2xx requests; keep all 4xx and 5xx.
• Correlate logs with traces via request_id or x-request-id.

A centralized log structure lets you trace a single request from the gateway all the way to the backend, complementing metrics and traces.

#SLO, SLI, and Alerting

#Defining SLIs and SLOs

An SLI is a quantitative measure of service quality; an SLO is the agreed target. For the mesh, common SLIs:

• Availability: the percentage of requests without errors, for example a 99.9 percent target.
• Latency: p95 or p99 latency below a certain threshold.
• Throughput: requests per second.

An example latency SLI with Prometheus:

histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket{reporter="destination",destination_service="productpage.default.svc.cluster.local"}[5m])) by (le))

histogram_quantile(0.99, ...) computes productpage's p99 latency over five minutes. This metric is used to determine whether the SLO is met.

#Error-Budget-Based Alerting

The golden rule of SLOs: alert when the error budget is running low, not when a single request fails. The error budget is 100 minus the SLO. If the SLO is 99.9 percent, the budget is 0.1 percent per month. Healthy alerting:

• Page only when the budget is nearly exhausted or exhausted.
• Dashboards for normal daily trends.
• Runbooks for every alert that can page.

This principle prevents alert fatigue and ensures the pages that come in really need a human response.

#Summary

Episode 20 prepared observability for large scale: managing metric cardinality, using smart trace sampling, centralized log aggregation, and building measurable SLI-based SLOs with error-budget alerting.

Key takeaways:

• High cardinality is Prometheus's enemy; drop unused dimensions.
• Recording rules and downsampling reduce storage load.
• Tail-based or prioritized sampling is more accurate than random.
• useRequestIdForTraceSampling keeps sampling consistent across hops.
• Centralized log aggregation enables correlation with traces.
• Measure SLIs first, then set SLOs from the real baseline.
• Alert from the error budget, not from a single failure.

In the next episode, episode 21, we will protect the mesh's long journey: upgrade, backup, and disaster recovery — a safe upgrade path with canaries, rollback strategies, backing up and restoring Istio configuration, and cluster recovery.