Learning GitOps - FluxCD - Canary Deployments

#Introduction

In episode 15 you got to know Flagger: its architecture, how to install it, and its general workflow. Now it's time to dissect the most popular strategy — canary deployment — in depth.

In this episode 16 we'll discuss how Flagger shifts traffic bit by bit, how the Canary CRD is configured, how routing works across various providers, how metrics are analyzed, and what conditions trigger automatic rollback.

#The Canary Strategy

Canary deployment sends a small portion of traffic to the new version while the majority stays on the old version. If the metrics on that small traffic are healthy, the traffic share is increased gradually until the new version receives all the traffic.

The advantages of this strategy:

• Incremental traffic shifting — minimal impact per change
• Step-by-step progression — every stage is verified before continuing
• Metrics validation — decisions based on data, not assumptions
• Automatic rollback — a failure at any stage is immediately reverted

#Canary CRD Configuration

All strategies are controlled from a single object: the Canary. Here's an example of a full canary deployment configuration:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  service:
    port: 9898
    portName: http
  analysis:
    interval: 1m
    iterations: 5
    threshold: 99
    stepWeight: 10
    maxWeight: 50
    metrics:
      - name: request-success-rate
        threshold: 99
        interval: 1m
      - name: request-duration
        threshold: 500
        interval: 30s
    webhooks:
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 20 http://podinfo-canary.test:9898/"

#Explanation of the Main Fields

#Traffic Routing

Flagger doesn't shift traffic itself — it configures the provider's routing resources. Three common approaches:

An example Istio VirtualService managed by Flagger:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: podinfo
  namespace: test
spec:
  hosts:
    - podinfo.test
  http:
    - route:
        - destination:
            host: podinfo
          weight: 100
        - destination:
            host: podinfo-canary
          weight: 0

While the analysis runs, Flagger updates the weight gradually, for example from 100 to 0 to 90 to 10, then 80 to 20, and so on.

For NGINX, Flagger creates a canary Service and dynamically manages the annotations on the ingress such as nginx.ingress.kubernetes.io/canary and nginx.ingress.kubernetes.io/canary-weight.

#Metrics Analysis

Metrics analysis is the heart of canary deployment. By default Flagger uses two built-in metrics: request-success-rate and request-duration, both pulled from Prometheus.

#Prometheus Metrics

Flagger reads HTTP metrics from Prometheus with a query targeting the canary and primary traffic. For Pods that don't produce HTTP metrics, add a Prometheus exporter to the application.

#Request Success Rate

This metric measures the percentage of successful requests. The default threshold of 99 means 99 percent of requests must succeed during analysis:

metrics:
  - name: request-success-rate
    threshold: 99
    interval: 1m

#Request Duration

This metric measures latency. The threshold value is in milliseconds:

metrics:
  - name: request-duration
    threshold: 500
    interval: 30s

#Custom Metrics and MetricTemplate

For more specific business or technical metrics, use a MetricTemplate and reference it inside metrics:

analysis:
  metrics:
    - name: not-found-percentage
      templateRef:
        name: not-found-percentage
        namespace: test
      thresholdRange:
        max: 5
      interval: 1m

#Progressive Traffic Shifting

The combination of stepWeight, maxWeight, iterations, and interval determines the traffic shifting curve. With stepWeight: 10 and maxWeight: 50, the traffic sequence tested is: 10 percent, 20 percent, up to 50 percent. If maxWeight is reached and all metrics are healthy, Flagger immediately promotes the new version to 100 percent.

#Rollback Conditions

Rollback happens automatically when one of the following conditions is met:

• Metric threshold breaches — success rate or duration crosses the threshold
• Failed checks threshold — the number of failed iterations crosses the threshold percent
• Webhook failure — the load tester or another hook returns an error
• Manual rollback — kubectl is used to stop the analysis

When rollback is triggered, Flagger returns all the traffic to the primary version and marks the canary as failed:

kubectl get canary podinfo -n test
kubectl describe canary podinfo -n test

The command below can be used for manual rollback or to restart the analysis:

kubectl -n test patch canary podinfo \
  --type merge \
  -p '{"status":{"phase":"Failed"}}'

#Closing

Episode 16 dissected canary deployment as a gradual, data-driven deployment strategy.

The key takeaways:

• The Canary CRD controls the whole strategy: target, service, analysis, metrics, and webhooks.
• Traffic shifting is managed through stepWeight, maxWeight, iterations, and interval.
• Routing is delegated to the provider: Istio VirtualService, Linkerd TrafficSplit, NGINX annotations, or Gateway API HTTPRoute.
• Prometheus metrics like success rate and duration are the basis for promote or rollback decisions.
• Automatic rollback happens when a metric threshold is breached, a webhook fails, or it's requested manually.

In the next episode, episode 17, we'll discuss Blue/Green and A/B Testing — two other progressive strategies: blue-green with traffic mirroring and instant switching, and header and cookie-based A/B testing for user experiments. See you!