Learn Envoy Proxy - GitOps, CI/CD & Configuration Management

#Introduction

Envoy config is code — and code must be managed like code. Episode 20 covers GitOps, CI/CD, and configuration management: storing Envoy config in Git as the single source of truth, validating config automatically in the pipeline, and rolling out changes safely.

Many production incidents are caused by untested config changes. With the flow you learn in this episode, every config change goes through review, validation, and controlled deployment — exactly like a code change.

#Managing Envoy Config in GitOps

#Config as Code

The GitOps principle is simple: all Envoy config is stored in a Git repository, deployed by automated tooling, and the actual state is always pulled toward the desired state. No SSHing into servers to edit config.

A recommended repository structure:

envoy-configs/
  base/
    bootstrap.yaml
  overlays/
    staging/
      bootstrap.yaml
    production/
      bootstrap.yaml
  clusters/
    orders-service.yaml
  README.md

The overlays/ directory separates per-environment config differences. Tools like Kustomize or Helm make this pattern easy, and Argo CD or Flux keeps clusters in sync with the repository.

#Differences Between Environments

Usually only a few things differ between environments: control plane addresses, logging levels, and certificate sources. Separate what changes from what stays the same — don't copy the entire config.

#Validating Config in CI/CD

#The Envoy Validation Command

Envoy has a built-in validation mode that's perfect for pipelines:

envoy --mode validate -c envoy.yaml
docker run --rm -v $(pwd):/etc/envoy \
  envoyproxy/envoy:v1.31.0 envoy --mode validate -c /etc/envoy/envoy.yaml

The envoy --mode validate -c envoy.yaml command checks the entire config without running the proxy. Validation reads the YAML, verifies all filter types and names, and checks references between resources. An OK output means the config is ready to deploy.

#Validation in GitHub Actions

Put validation in the pipeline as a mandatory step:

name: validate-envoy-config
on:
  pull_request:
    paths:
      - "configs/**"
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate Envoy config
        run: |
          for f in configs/*.yaml; do
            docker run --rm \
              -v "$PWD":/etc/envoy \
              envoyproxy/envoy:v1.31.0 \
              envoy --mode validate -c "/etc/envoy/$f"
          done

The validate-envoy-config workflow runs validation on every pull request touching the configs directory. If any config is broken, the PR can't be merged — errors are caught before reaching production.

#Testing Config with a Real Runner

Validation isn't enough for config that's syntactically correct but logically wrong (for example a route to a cluster that doesn't exist). For deeper testing:

• Run Envoy with that config in CI and send test requests.
• Compare behavior with the previous config.
• Use integration tests that simulate real traffic.

#Rolling Updates and Change Management

#Config Rollout Strategy

Config changes must be rolled out, not applied to every instance at once. The common pattern:

1. Apply to a single staging instance.
2. Validate and run smoke tests.
3. Roll out to a production subset.
4. Monitor metrics and access logs.
5. Complete the whole fleet.

#Kubernetes Rolling Updates with ConfigMap

On Kubernetes, config changes usually go through a ConfigMap:

kubectl apply -f configs/production/orders-envoy.yaml
kubectl rollout status deployment/orders -n prod

The kubectl apply command changes the ConfigMap, and the restarted deployment loads the new config. For safer rollouts, combine it with the Envoy health checks prepared in episode 7 — a pod that fails readiness doesn't receive traffic.

#Feature Flags and Rollback

The fastest way to handle a problematic config is rollback:

git revert HEAD
kubectl rollout undo deployment/orders -n prod

The git revert HEAD command restores the config in the repo, and kubectl rollout undo returns the pods to the previous version. This rollback speed is the main value of GitOps: every change can be traced and undone precisely.

#Change Audit Trail

Because all changes go through Git, every config has a history: who changed what, when, and which review approved it. This isn't just compliance — the audit trail is the first debugging tool when a config causes problems.

#Building a Safe Pipeline

#Pre-Merge Checklist

Here are the checks that must pass before a config is merged:

• envoy --mode validate for all environments.
• Diff staging and production config to catch accidental differences.
• Smoke test in the staging environment.
• Approval from a reviewer who understands the config's impact.

#A Complete Flow Example

commit -> validate -> review -> merge -> sync (Argo CD) -> health check -> done

This commit -> validate -> review -> merge flow ensures every change reaches production Envoy with two layers of safety: the pipeline and human review. Combine it with the dynamic xDS from episode 9, and config changes can even be applied without any restart.

#Closing

Episode 20 organized how you professionally manage Envoy config: GitOps as the source of truth, automatic validation in CI/CD, and rolling updates with fast rollback.

Key takeaways:

• Store all Envoy config in Git as the single source of truth.
• envoy --mode validate -c checks config without running the proxy.
• Run validation in CI on every pull request touching config.
• Roll changes out gradually and monitor metrics at each stage.
• GitOps makes every change traceable and quickly rollback-able.
• Combine GitOps with xDS for changes without restarts.

In the next episode, episode 21, we'll discuss observability at scale and SLOs — monitoring high-cardinality Envoy metrics, trace sampling and log aggregation strategies, and SLOs and SLIs for the proxy layer.