Learn Envoy Proxy - Observability at Scale & SLOs

#Introduction

In episode 11 you connected Envoy to Prometheus, tracing, and access logs. Episode 21 answers the question that comes up as scale grows: how do you monitor thousands of Envoys without blowing up your observability storage, and how do you turn that data into a measurable promise — an SLO (Service Level Objective).

You'll learn to handle high-cardinality metrics, choose sampling strategies for tracing, aggregate logs without losing context, and design SLI/SLOs for the proxy layer that genuinely reflect user experience.

#Monitoring High-Cardinality Envoy Metrics

#Where Cardinality Comes From

Envoy's per-listener and per-cluster metrics are already many, but cardinality explodes when dynamic labels enter — for example per-request header values, many virtual hosts, or custom tags from stats_tags. Every new label combination means a new time series in Prometheus.

stats_config:
  stats_tags:
    - tag_name: cluster
      regex: "^cluster\\.(.+?)\\.(upstream|membership)\\."
    - tag_name: virtual_host
      regex: "^vhost\\.(.+?)\\."

The stats_tags rules determine which labels are extracted. The more tags you define, the higher the cardinality. Choose tags actually used in queries and dashboards — leave the rest as part of the metric name.

#Practices for Controlling Cardinality

Some habits that keep Prometheus healthy:

• Limit tags to a few key dimensions: cluster, listener, zone.
• Avoid putting request header values into labels.
• Use reasonable histogram_buckets, not overly fine ones.
• Watch the time series count per instance and stay within scrape targets.

curl -s localhost:9901/stats/prometheus | grep -c "^envoy"

The ^envoy grep counts the number of metric series one Envoy exports. Multiply by the number of instances to estimate Prometheus load. If the number starts getting out of hand, reevaluate the tags you defined.

#Metric Sampling

For metrics that are very large, sampling may be necessary. The concept: don't scrape every instance every interval; scrape a subset or reduce the interval. The trade-off between granularity and cost must be decided deliberately.

#Trace Sampling and Log Aggregation

#Tracing Sampling Strategies

Recording 100 percent of traces at large scale isn't realistic. Common sampling strategies:

• Head-based sampling: decide up front whether a trace is kept.
• Tail-based sampling: wait for the trace to finish, then decide based on interesting events (errors, high latency).
• Error-only sampling: always keep traces that contain errors.

tracing:
  http:
    name: envoy.tracers.opentelemetry
    typed_config:
      "@type": type.googleapis.com/envoy.tracers.opentelemetry.v3.OpenTelemetryConfig
      grpc_service:
        envoy_grpc:
          cluster_name: otel_collector
      service_name: envoy-gateway
      sampling_config:
        default_sampling_percentage: 10

The value default_sampling_percentage: 10 makes Envoy sample only 10 percent of requests for tracing. Start with a low percentage, then raise it only when investigation needs are truly high.

#Efficient Log Aggregation

Envoy access logs can be enormous. The keys to efficient aggregation:

• Log to stdout and let an agent (Fluent Bit, OpenTelemetry Collector) do the parsing.
• Store full logs in cold storage; query with limited indexes.
• Keep the log format stable so agent parsing doesn't change.
• Use the trace ID field to join logs and traces during investigation.

format: "%START_TIME% %DOWNSTREAM_REMOTE_ADDRESS% %REQ(X-REQUEST-ID)% %REQ(TRACEPARENT)% %RESPONSE_CODE% %RESPONSE_FLAGS% %DURATION%ms %UPSTREAM_CLUSTER% %UPSTREAM_HOST% %REQ(METHOD)% %REQ(PATH)%\n"

The %RESPONSE_FLAGS% and trace ID format above gives all the debugging information without burdening the agent with complex parsing. Structured logs are worth more than long logs.

#SLOs and SLIs for the Proxy Layer

#Defining SLIs

An SLI is a concrete measurement of service quality. For the proxy layer, common SLIs:

• Availability: the proportion of successful requests out of total.
• Latency: latency percentiles like p99.
• Throughput: requests per second processed.
• Error rate: the proportion of 5xx responses out of total.

#Defining SLOs

An SLO is a target chosen from an SLI, for example "p99 latency below 200ms 99.9 percent of the time". Make SLOs realistic — too strict and the team gets paged constantly; too loose and users aren't protected.

#Querying SLIs from Envoy Metrics

curl -s localhost:9901/stats/prometheus | grep "upstream_rq_time" | head -3
curl -s localhost:9901/stats/prometheus | grep "upstream_rq_5xx"

To calculate the error budget periodically, query Prometheus aggregates:

sum(rate(envoy_cluster_upstream_rq_5xx[5m])) by (envoy_cluster_name)
  / sum(rate(envoy_cluster_upstream_rq_total[5m])) by (envoy_cluster_name)

The PromQL query envoy_cluster_upstream_rq_5xx computes the proportion of 5xx errors per cluster over 5 minutes. The error budget is calculated by comparing the actual SLI against the SLO target over the running period.

#SLOs in a Service Mesh

In a mesh ecosystem, SLOs also cover services as a whole, but the proxy layer has a special role: mTLS, retries, and routing affect availability from the infrastructure side. Monitor proxy SLIs separately from application SLIs so you can distinguish "proxy problem" from "application problem".

#Error Budget and Decision Making

#What to Do When the Budget Runs Out

An error budget isn't just a number — it's a decision tool:

• When the budget is safe: free to deploy new features and config changes.
• When the budget is thinning: slow down releases, focus on reliability.
• When the budget is exhausted: stop releases, fix the cause, reevaluate the SLO.

A good SLO turns "feeling safe" decisions into data-driven ones. For the proxy layer, this means combining Envoy metrics with the release process built in episode 20.

curl -s localhost:9901/stats/prometheus | grep "envoy_cluster_upstream_rq_5xx"

Monitor envoy_cluster_upstream_rq_5xx periodically. A spike approaching the budget limit is a signal to hold changes until the trend drops.

#Closing

Episode 21 closed out the observability pillar: managing metric cardinality, sampling and aggregation strategies, and designing SLO/SLIs that turn Envoy data into operational decisions.

Key takeaways:

• Metric cardinality explodes through dynamic labels; limit it with deliberate stats_tags.
• A 10 percent trace sample is a reasonable starting point; raise it only when needed.
• A stable log format makes agent parsing and aggregation easy.
• Proxy SLIs: availability, percentile latency, throughput, error rate.
• The error budget is a decision tool, not just a number.
• Distinguish proxy SLIs from application SLIs so root causes are easy to find.

In the next episode, episode 22, we'll discuss production hardening and best practices — a security hardening checklist, an operational checklist with runbooks and debugging tools, and Envoy upgrade considerations with fallback strategies.