Learning GitOps - FluxCD - Blue/Green & A/B Testing

#Introduction

In episode 16 you dissected canary deployment: gradual traffic shifting, metrics analysis, and automatic rollback. But not every release fits that gradual model. For big releases or user experiments, there are other strategies: blue/green and A/B testing.

In this episode 17 we'll cover both, plus two supporting practices that often accompany them: load testing before promotion and conformance testing as a quality gate.

#Blue/Green Deployments

Blue/green is a strategy that maintains two full environments in parallel: blue is the old version, green is the new version. All traffic stays on blue while green is validated, then the traffic is switched to green instantly.

The advantages of blue/green:

• Traffic mirroring — the new version can be tested with copied production traffic
• Manual promotion — the traffic switch can wait for a human decision
• Instant rollback — returning to blue only takes one switch
• Zero-downtime updates — no service gap during promotion

In Flagger, blue/green is configured with stepWeight and maxWeight set to 100. The analysis runs for several iterations without shifting traffic, then all traffic is switched at once:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  service:
    port: 9898
  analysis:
    interval: 1m
    iterations: 10
    stepWeight: 100
    maxWeight: 100
    metrics:
      - name: request-success-rate
        threshold: 99
        interval: 1m

With this configuration, Flagger creates the green environment, runs the analysis iterations times, and only switches traffic after all metrics are healthy. Because the whole traffic moves at once, make sure the green environment has been tested with adequate load before promotion.

#A/B Testing

A/B testing differs from canary: traffic isn't split by percentage, but by request characteristics such as a header or cookie. This lets a specific group of users try the new version while the rest stay on the old version.

Common uses:

• Header-based routing — a custom header, e.g. x-canary: insider, determines the experiment group
• Cookie-based routing — users with a specific cookie are directed to the new version
• User segmentation — selecting a subset of real users to test a feature
• Feature flag integration — combining routing with feature flag logic in the application
• Experimental traffic — measuring the real impact before a broad release

In Flagger, A/B testing is configured with stepWeight: 0 and match rules:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  service:
    port: 9898
  analysis:
    interval: 1m
    iterations: 10
    stepWeight: 0
    match:
      - headers:
          x-canary:
            exact: "insider"
    metrics:
      - name: request-success-rate
        threshold: 99
        interval: 1m

If match uses a cookie, the structure used is cookies with the same rules. During the iterations, Flagger directs requests that match the match condition to the canary version and evaluates the metrics on that traffic. Promotion happens if all iterations pass.

#Load Testing Integration

Blue/green and A/B only mean something if the new version is truly tested under load. Flagger integrates a load tester through webhooks on analysis.webhooks.

#Pre-Promotion Load Tests

The load test webhook runs on every iteration to make sure the canary version can withstand the traffic:

analysis:
  interval: 1m
  iterations: 10
  webhooks:
    - name: load-test
      url: http://flagger-loadtester.test/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -c 50 -q 20 http://podinfo-canary.test:9898/"

#Webhook Tests

Other webhook types can be used for additional verification, for example checking a specific endpoint, running a smoke test, or validating the response to a particular payload.

#Concurrency Testing

The parameters in the load tester command set the concurrency level, for example -c 50 means 50 parallel connections. Increase this value gradually to find the weak point before production traffic finds it.

#Performance Baselines

Record the load test results of the old version as a baseline. Compare the new version's latency and error rate against that baseline as one of the pass criteria.

#Conformance Testing

Beyond load, a release also needs to pass conformance tests against agreed criteria. Flagger supports various approaches for this.

#Helm Test Integration

For applications managed by Helm, run helm test as a hook before promotion. A hook that returns an error will fail the analysis and trigger a rollback:

analysis:
  interval: 1m
  iterations: 5
  webhooks:
    - name: helm-test
      type: rollout
      url: http://flagger-hook.test/
      timeout: 30s
      metadata:
        cmd: "helm test podinfo -n test"

#Custom Test Jobs

For more complex testing, run a custom Kubernetes job that executes the testing scenario, then use a webhook to wait for the result before the analysis continues.

#Acceptance Criteria and the Promotion Gate

Combine everything into a promotion gate: the load test, helm test, and additional webhooks must pass on every iteration. If any of them fails, Flagger holds the promotion and the traffic returns to the old version. These criteria are documented directly in the Canary CRD, so the whole team can see what a release requires.

#Closing

Episode 17 closes the progressive delivery phase with two strategies that complement canary: blue/green and A/B testing, plus strengthening through load testing and conformance testing.

The key takeaways:

• Blue/green uses stepWeight: 100 and maxWeight: 100, maintaining two full environments with instant rollback.
• A/B testing uses stepWeight: 0 with header or cookie-based match for user experiments.
• Load testing runs through a webhook on every iteration to validate the new version's capacity.
• Conformance testing combines helm test, custom jobs, and acceptance criteria as a promotion gate.
• All strategies are managed in a single Canary CRD, so the release policy is documented as code.

In the next episode, episode 18, we'll discuss Service Mesh Integration — how Istio, Linkerd, and AWS App Mesh work together with FluxCD and Flagger for traffic management, observability, and canary visualization. See you!