Learn Istio - Observability (Metrics, Logs & Tracing)

#Introduction

From episode 5 to 7, you configured traffic and resilience. But how do you know this machine is working well? The answer: observability. Episode 8 builds the foundation for seeing what happens inside the mesh — metrics, logs, and traces — along with the dashboards that make it easy to read.

This is also the episode you will keep referring back to in episode 20. Treat this one as the telemetry foundation, and episode 20 later as its optimization.

#Telemetry Pipeline: Telemetry v2

Modern Istio uses Telemetry v2, a metrics model built on Envoy standard metrics with Prometheus scraping merged in. The sidecar exposes metrics on port 15020:

kubectl exec -it productpage-abc123 -c istio-proxy -- curl localhost:15020/stats/prometheus | head -20

curl localhost:15020/stats/prometheus shows Prometheus metrics in text format. The ones that matter most for traffic, rather than resources, are the metrics prefixed with istio_:

• istio_requests_total: the number of requests per service pair, including the response_code and response_flags labels.
• istio_request_duration_milliseconds: a histogram of request duration.
• istio_request_bytes and istio_response_bytes: payload sizes.

An example query for monitoring the productpage error rate:

sum(rate(istio_requests_total{reporter="destination",destination_service="productpage.default.svc.cluster.local",response_code=~"5.."}[5m]))
/
sum(rate(istio_requests_total{reporter="destination",destination_service="productpage.default.svc.cluster.local"}[5m]))

The equation above calculates the percentage of 5xx requests against total requests in five minutes. reporter="destination" means the metrics are recorded on the receiving side, so you see the backend experience rather than the client perspective.

#Distributed Tracing

#Header Propagation and Sampling

Traces stitch together the journey of one request across many services. To keep a trace intact, Envoy injects propagation headers into every request. Istio supports standard formats like x-request-id, b3, and traceparent. The key point: the trace context must be passed from one service to the next, so applications must forward those headers downstream.

Sampling controls what percentage of requests gets traced. Configure it through the Telemetry API:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
  - providers:
    - name: otel
    randomSamplingPercentage: 10

randomSamplingPercentage: 10 traces 10 percent of requests. In high-volume production, 100 percent sampling is usually too expensive — episode 20 will cover advanced sampling strategies.

#Viewing Traces

Once Jaeger or Tempo is installed and traces are received, you can see the waterfall: the duration of each span, which service is slow, and where failures happen. Istio generates a span for every Envoy hop; applications can add custom spans through the OpenTelemetry SDK.

#Access Logs

Envoy access logs record every request that passes through the sidecar. Enable them through the Telemetry API:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
  - providers:
    - name: envoy

With the envoy provider, default-format logs appear in the istio-proxy container and can be read:

kubectl logs productpage-abc123 -c istio-proxy | tail -5

kubectl logs ... -c istio-proxy shows the access log of every request. For automatic parsing in production, switch the format to JSON and point it at your log pipeline — we will cover log aggregation in episode 20.

#Dashboards: Kiali and Grafana

#Kiali for the Service Graph

Kiali is the Istio-specific dashboard. It builds the service graph from mesh metrics and configuration:

kubectl apply -f samples/addons/kiali.yaml
kubectl port-forward -n istio-system svc/kiali 20001:20001

kubectl port-forward ... svc/kiali 20001:20001 opens Kiali at localhost:20001. From here you can see the traffic graph between services, configuration validation, and per-edge request metrics — a tool that really helps in episodes 9 and 19.

#Grafana for Dashboards

Grafana shows ready-to-use Prometheus metric dashboards:

kubectl port-forward -n istio-system svc/grafana 3000:3000

The Istio dashboards present Overview, Mesh, Workload, and Service in one place. All the metrics we discussed are visualized without writing manual queries.

#Summary

Episode 8 brought the mesh's eyes to life: Telemetry v2 with Prometheus metrics on port 15020, distributed tracing with header propagation and sampling, Envoy access logs, and Kiali and Grafana dashboards that present everything in one place.

Key takeaways:

• Key metrics: istio_requests_total, istio_request_duration_milliseconds.
• reporter="destination" shows the backend perspective.
• Traces need correct header propagation to stay connected across services.
• Sampling controls tracing cost; 10 percent is enough for many cases.
• Envoy access logs are enabled via the Telemetry API and read from the istio-proxy container.
• Kiali for the service graph; Grafana for metric dashboards.
• Always verify with real metrics, not assumptions.

In the next episode, episode 9, we will keep the configuration healthy: configuration management and validation — istioctl analyze, how to read validation results, EnvoyFilter and its best practices, and GitOps patterns with ArgoCD or Flux.