Learning GitOps - FluxCD - Testing Strategies

#Introduction

In episode 27 you built a CI/CD pipeline that flows automatically from code commit to deployment through Git. Changes can now reach production at high speed — which means mistakes can also reach production at the same speed. The question that arises is: how do we make sure of the quality before a change touches the real cluster?

Speed without a safety net is a recipe for incidents. A deployment that's run smoothly for months can suddenly break production because of one manifest with a wrong schema, one violated policy, or one service that was forgotten to migrate. Testing in the GitOps world isn't only about testing the application — it's about testing the manifests, the deployment process, and the resilience of the cluster itself.

In this episode we'll build a layered testing strategy for the Flux stack: manifest validation with linting and policy, dry-run testing with flux diff, PR-based preview environments, post-deploy integration testing, and chaos engineering to prove the cluster can survive failures.

#Manifest Validation

#YAML Linting

The most basic and most common mistake is invalid YAML — wrong indentation, tabs in the middle of spaces, or a broken structure. Broken YAML will make the whole render fail. Linting is the first net that catches it. yamllint checks syntax and style rules:

yamllint clusters/ apps/

Run linting on every pull request before manifests are allowed through the pipeline. It's cheap, fast, and prevents a large number of trivial problems at later stages.

#Kubernetes Manifest Validation

Valid YAML isn't necessarily valid as a Kubernetes manifest. Object schemas differ between API versions, and a typo in a field can make an object rejected or created with unexpected behavior. kubeconform validates manifests against the official Kubernetes schema:

kubeconform -strict -summary \
  -schema-location default \
  clusters/ apps/

Schema validation catches problems like deprecated API versions or wrongly named fields. To make sure the Flux CRDs are also valid, add a dedicated schema for the Flux custom resources.

#Policy Validation with OPA and Kyverno

Schema validation only checks structure, not policy. Policies like "every container must have resource limits" or "images must not come from unknown registries" need a policy engine. Two common tools:

• Kyverno — Kubernetes-native policies written as ClusterPolicy. Suitable if you want policies written in simple YAML and run as an admission controller in the cluster.
• OPA Gatekeeper — the OPA (Open Policy Agent) implementation for Kubernetes. Policies are written in Rego and enforced through ConstraintTemplate and Constraint.

An example Kyverno policy requiring resource limits:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resources
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-limits
      match:
        any:
          - resources:
              kinds:
                - Deployment
      validate:
        message: "Container must have resource limits"
        pattern:
          spec:
            template:
              spec:
                containers:
                  - resources:
                      limits:
                        cpu: "?*"

With validationFailureAction: Enforce, a violating deployment is rejected before it enters the cluster. Run the same policy in CI (for example with kyverno test) and in the cluster as an admission controller for a double layer.

#Dry-Run Testing

#flux diff

Before a change is applied, we need to know exactly what will change in the cluster. flux diff shows the difference between the manifests in Git and the current cluster state, without applying anything:

flux diff kustomization apps \
  --path ./apps \
  --kustomization-file ./clusters/production/apps.yaml

flux diff marks the objects that will be created, changed, or deleted. This is the main tool for impact analysis before a pull request is merged.

#Server-Side Dry Run

The second dry-run layer asks the cluster to compute the result server-side. kubectl apply --dry-run=server sends the manifests to the API server which does schema and admission webhook validation without saving the changes:

kubectl apply --dry-run=server -f rendered-manifest.yaml

A server-side dry run catches problems that only the server can know — such as conflicts with existing resources, rejecting admission webhooks, or exceeded quotas. Run it after flux build produces the final manifests.

#Impact Analysis

Combine both dry-runs for a complete impact analysis:

1. flux build to produce the final manifests from Git.
2. flux diff to see the differences against the cluster state.
3. kubectl apply --dry-run=server for server validation.
4. Human review of the diff output — especially objects that will be deleted, because deletion is the riskiest change.

A documented impact analysis also helps the review team: proof that the proposed change has been verified, not just a claim.

#Preview Environments

#PR-Based Environments

A preview environment is a temporary cluster or namespace created for each pull request — a place where changes are tested in real conditions before the merge. A common flow:

1. A PR is created — the workflow creates a namespace with a unique name per PR.
2. Manifests are deployed to that namespace.
3. Testers try the application at a preview URL.
4. The PR is merged or closed — the namespace is cleaned up.

The key is a unique namespace per PR, so several PRs can be tested at the same time without conflicts.

#Ephemeral Clusters

For a larger scale, a preview can be a full ephemeral cluster — created automatically on a PR, destroyed when the PR finishes. This approach is closest to the real production conditions, including the installed operators and policies. But its cost is higher and the wait time is longer, so use it wisely:

• Namespace per PR for ordinary applications — cheap and fast.
• Ephemeral cluster for infrastructure changes or end-to-end testing that needs full isolation.

#Cleanup Automation

A preview environment that's never cleaned up piles up resources and cost. Cleanup automation must be part of the workflow from the start:

name: cleanup-preview
on:
  pull_request:
    types: [closed]
jobs:
  cleanup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          flux delete kustomization preview-${{ github.event.number }} \
            --silent --force || true
          kubectl delete namespace preview-${{ github.event.pull_request.number }} \
            --ignore-not-found

Note the use of --force so Flux deletes the dependent resources that follow it. Automatic cleanup keeps the cluster tidy without relying on team discipline.

#Integration Testing

#Post-Deploy Testing

A successful deployment doesn't mean a working application. Post-deploy integration testing verifies that the newly deployed application actually works together with its dependencies. Start by explicitly waiting for the Ready condition:

kubectl wait --for=condition=Ready pod \
  -l app.kubernetes.io/name=checkout --timeout=120s

Then verify real integration — call the endpoint, check the database connection, and make sure the service routing works.

#Smoke Tests

A smoke test is a quick test that makes sure the main services work after a deployment — not a thorough test, but an early alarm. An example of a simple smoke test:

curl -sf http://checkout.apps/healthz && echo "healthy"

If the smoke test fails, the pipeline marks the deployment as problematic and triggers a rollback or investigation. Speed is the priority: a smoke test must finish in seconds so it can run on every deployment.

#End-to-End Tests

An end-to-end (E2E) test runs scenarios from the end user's perspective: login, checkout, and a complete flow across many services. Tools like k6 for performance or Playwright/Cypress for UI run after the deployment finishes and the smoke test passes. E2E is the most expensive layer — run it in preview environments or staging, not on every production deployment.

#Chaos Engineering

#Failure Injection

Chaos engineering is a deliberate resilience test: breaking something in a controlled environment to prove the system can survive. The goal isn't to damage production, but to find weaknesses before a real incident finds them. An example of a simple and safe failure injection:

kubectl delete pod -l app=checkout --wait=false

Flux should restore the ReplicaSet to the desired count. Also test more complex injections: shut down a node, disrupt network between services, or make a registry inaccessible. Tools like Litmus or Chaos Mesh automate these injections with schedulable scenarios.

#Recovery Testing

After a failure is injected, the next important part is verifying the recovery — including Flux's role in it. The question answered: does Flux really restore the state to the desired state? GitOps-relevant recovery tests:

• Delete a Flux-managed resource and wait for it to reappear.
• Change a manifest in the cluster manually and see Flux return it to Git.
• Cut the Git connection temporarily and make sure the last deployment keeps running (no drift).

kubectl delete deployment checkout -n apps
flux reconcile kustomization apps
kubectl get deployment checkout -n apps

#Resilience Validation

Chaos engineering results must become metrics, not anecdotes. Record every experiment, its impact, and whether the system recovered in the expected time. Validated resilience gives confidence to deploy faster: we know the system's limits, not guess them.

Start with chaos in staging, expand slowly to production with scenarios whose impact is limited (for example deleting one Pod in a non-critical environment), and always provide a way to stop the experiment quickly.

#Closing

In this episode 28 you built a layered testing strategy for the Flux stack: manifest validation with yamllint, kubeconform, and OPA/Kyverno policy, dry-run testing with flux diff and server-side dry run for impact analysis, PR-based preview environments with cleanup automation, post-deploy integration testing with smoke and end-to-end tests, and chaos engineering to validate the cluster's resilience and recovery.

The key takeaways:

• Testing starts from the manifests, not from the application — lint, validate the schema, and enforce policy before deployment.
• flux diff and dry-run are the PR safety net — they show the real impact without changing the cluster.
• Preview environments must have a lifecycle — created per PR and deleted automatically when the PR finishes.
• Graduated testing adjusts cost and frequency — smoke tests on every deployment, E2E in staging.
• Chaos engineering proves, not guesses, resilience — every experiment is recorded and measured.

Now your code, manifests, and cluster are tested. But there's one layer that hasn't fully entered Git: infrastructure itself. In the next episode, episode 29, we'll discuss Infrastructure as Code with Terraform — managing clusters and infrastructure with Terraform, Flux provider integration and bootstrap via Terraform, infrastructure layers, and GitOps for infrastructure with Crossplane, the Terraform Controller, and Atlantis. Keep up the momentum!