Learn Envoy Proxy - Access Logging & Basic Observability

#Introduction

Envoy is a proxy that "talks" through data: every passing request can be logged, measured, and monitored. Episode 7 opens the world of access logging and basic observability — how to enable per-request logs, build a log format with useful metadata, make sure Envoy itself is healthy through health check endpoints, and expose metrics in Prometheus format.

This is the observability foundation that will be deepened in episodes 11 and 21. After this episode, you don't just have a working proxy — you have one you can talk to.

#Enabling Access Logs

#Access Log in the HTTP Connection Manager

Access logs are defined inside http_connection_manager:

Mengaktifkan access log ke stdout

listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
      - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              access_log:
                - name: envoy.access_loggers.file
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                    path: /dev/stdout
                    format: "[%START_TIME%] %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %RESPONSE_CODE% %DURATION%ms\n"
              route_config:
                name: local_route
                virtual_hosts:
                  - name: local_service
                    domains:
                      - "*"
                    routes:
                      - match:
                          prefix: "/"
                        route:
                          cluster: api_service
              http_filters:
                - name: envoy.filters.http.router
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

The access_log section adds one file logger writing to /dev/stdout with a custom format. This format uses Envoy operators like %START_TIME%, %RESPONSE_CODE%, and %DURATION% to print metadata for each request.

#Watching Access Logs Live

Every request passing through Envoy is now recorded. Send a few requests, then observe the logs:

Membaca access log kontainer

curl -s -H "Host: localhost" http://localhost:10000/health
docker logs envoy-obs 2>&1 | tail -5

The docker logs envoy-obs command shows the access logs written to the container's stdout. Each line represents one request, complete with response code and duration.

#Log Format and Request Metadata

#Frequently Used Operators

Envoy has hundreds of format operators. The most useful for basic observability:

%REQ(HEADER)?:DEFAULT% — the value of one request header.
%RESP(HEADER)?:DEFAULT% — the value of one response header.
%START_TIME% — when the request started.
%RESPONSE_CODE% — the response status code.
%DURATION% — the request duration in milliseconds.
%UPSTREAM_CLUSTER% — the cluster that handled the request.
%DOWNSTREAM_REMOTE_ADDRESS% — the client address.

Format log yang informatif

format: "%DOWNSTREAM_REMOTE_ADDRESS% %REQ(X-FORWARDED-FOR)% %START_TIME% \"%REQ(METHOD)% %REQ(PATH)%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %DURATION%ms %UPSTREAM_HOST%\n"

The %RESPONSE_FLAGS% format operator records how a request was processed, for example URX for retries or UF for upstream failure. These flags are invaluable when debugging, and will come back in episode 21.

#Global Format Configuration

The same format can be set once for all listeners via admin.access_log in the bootstrap, so you don't have to rewrite the format in every listener.

#Health Checks for Envoy Itself

#Liveness and Readiness

For platforms like Kubernetes, Envoy needs to expose its own health check endpoints. Use the admin interface as the target:

Endpoints health check Envoy

curl -s localhost:9901/ready
curl -s localhost:9901/livez
curl -s localhost:9901/healthcheck/fail

The endpoint local:9901/ready returns 200 when Envoy is ready to accept traffic, and local:9901/livez indicates the process is still alive. The healthcheck/fail command deliberately marks Envoy unhealthy — a technique orchestration platforms use to drain traffic before shutdown.

#Configuring in Kubernetes

In Kubernetes, these probes become livenessProbe and readinessProbe on the pod:

Probe Kubernetes untuk Envoy

readinessProbe:
  httpGet:
    path: /ready
    port: 9901
  initialDelaySeconds: 5
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /livez
    port: 9901
  initialDelaySeconds: 10
  periodSeconds: 10

The readinessProbe configuration ensures the Envoy pod doesn't receive traffic before it's truly ready, while livenessProbe makes the kubelet restart a stuck pod.

#Metrics in Prometheus Format

#Exposing Metrics

Envoy's admin interface already provides metrics in Prometheus format at a single endpoint:

Menarik metrics Prometheus

curl -s localhost:9901/stats/prometheus | head -20
curl -s localhost:9901/stats/prometheus | grep "^envoy_cluster" | head -5

The endpoint stats/prometheus displays all metrics in a format Prometheus can scrape directly. envoy_cluster_* metrics show per-cluster status: healthy endpoints, requests, and latency.

#Adding a Scrape Config

For Prometheus to collect data periodically, add the following job in prometheus.yml:

Job Prometheus untuk Envoy

scrape_configs:
  - job_name: envoy
    metrics_path: /stats/prometheus
    static_configs:
      - targets:
          - envoy-1:9901
          - envoy-2:9901

The target envoy-1:9901 tells Prometheus where to find Envoy's metrics endpoint. With this single job, all Envoy metrics flow into your monitoring system and can be charted in Grafana.

#Filtering Metrics

Because Envoy has so many metrics, use filters in the admin:

Filter metric tertentu

curl -s localhost:9901/stats?filter=envoy_cluster_api_service.upstream_cx_total
curl -s localhost:9901/stats/prometheus?filter=envoy_cluster_api_service

The filter=envoy_cluster_api_service parameter narrows the output to a single cluster. In production, filtering is important to avoid the parsing burden of thousands of metric lines on every scrape.

#Closing

Episode 7 gave you Envoy's first observability senses: per-request access logs with a configurable format, health check endpoints for orchestration, and Prometheus metrics ready to be scraped.

Key takeaways:

Access logs are defined in http_connection_manager via the access_log block.
Format operators like %REQ(...)%, %RESPONSE_CODE%, and %DURATION% shape the log.
%RESPONSE_FLAGS% records details of how a request was processed.
local:9901/ready and local:9901/livez for Envoy's own health checks.
stats/prometheus exposes all metrics in Prometheus format.
Add a simple Prometheus job so metrics are collected automatically.

In the next episode, episode 8, we'll discuss advanced Envoy filter chains — HTTP filters like ext_authz and the gRPC JSON transcoder, TCP filters like tcp_proxy, and the correct filter ordering and matching.