Learning GitOps - FluxCD - Production Checklist & Best Practices

#Introduction

In episode 33 you moved systems to FluxCD with a safe migration strategy. Now it's time to make sure everything is truly ready for production. In this episode 34 you'll learn the production checklist & best practices: a pre-production checklist from HA to DR, daily operational practices, Git repository governance, and the pitfalls that most often trip up teams.

#Pre-Production Checklist

#High Availability and RBAC

Make sure the Flux controllers run with more than one replica and have adequate resource limits. Check that all RBAC policies only grant minimal access, and that flux-system doesn't use the default Service Account:

Verify Flux controller readiness

kubectl get pods -n flux-system -o wide
flux check

#Encrypted Secrets

All secrets entering Git must be encrypted with SOPS or obtained from an external secret manager. Verify that no plaintext strings appear in the fleet repository:

Scan encrypted secrets with sops

sops --decrypt ./clusters/prod/secret.enc.yaml | yq '.data'

#Monitoring, Alerting, and Backup

Deploy kube-prometheus-stack to monitor the Flux controllers, and create alerts from the gotk_reconcile_condition and gotk_reconcile_duration metrics. Test the backup of Git repositories and cluster state periodically — not just prepare it.

#DR Plan and a Trained Team

Document the disaster recovery plan: how to rebuild the cluster from scratch with only flux bootstrap and the Git repository. Then train the team with monthly drills, because a DR document that's never tested is no better than having none.

Important

The beauty of GitOps is that recovery becomes simple: a new cluster + flux bootstrap + the fleet repository = a recovered environment. Test this statement regularly on a disposable cluster.

#Operational Practices

#Upgrades and Security Patches

Upgrade Flux following minor releases regularly with flux install --components-extra=... to update the controllers without losing existing resources. Monitor CVEs on controller images and dependency repositories.

#Resource and Capacity Monitoring

Monitor controller CPU/memory usage and gotk_reconcile_condition continuously. Do capacity planning before adding many applications, because every new Kustomization adds reconciliation load.

#Incidents and Runbooks

Build runbooks for common incidents: unsynced sources, failed health checks, and unreadable secrets. Document the flux get all, flux logs, and kubectl describe sequence as the starting point of an investigation.

Runbook: first steps of a Flux incident

flux get all --all-namespaces
flux events --all-namespaces --since=30m
kubectl describe kustomization flux-system -n flux-system

Tip

A good runbook answers the question "what do I do in the first 5 minutes". Write it as if the reader is panicking.

#Git Repository Best Practices

The fleet repository is the backbone of the whole system, so its governance must be as strict as production code:

Clear structure: folders per environment, per cluster, and per team that are easy to understand.
README: explain how to add applications, environments, and the rules of the game.
CODEOWNERS: define who approves changes in each part.
Branch protection: require reviews and status checks on main.
Commit convention: use Conventional Commits so the history and automatic releases stay readable.
PR template: request a consistent checklist on every change.

Branch protection in the repository settings

# Summary of the recommended rules
- require_pull_request_reviews: true
- required_approving_review_count: 1
- require_status_checks: true
- enforce_admins: true

Note

Treat the fleet repository like the most important production repository — it determines what runs in production. All the review policies usually applied to code, apply here too.

#Pitfalls to Avoid

Overly complex dependencies: Kustomizations referencing each other too deeply make tracing and debugging hard.
No health checks: without healthChecks, Flux considers a deployment successful even when its pod is crashing.
Inadequate testing: applying to production without validating the YAML render and schemas.
Poor secret management: plaintext secrets in Git or static credentials that are never rotated.
Insufficient monitoring: no alert for reconciliations failing repeatedly.
No rollback strategy: not knowing how to restore the environment if a change causes problems.
No documentation: configuration only lives in people's heads and becomes a knowledge silo.

A brief comparison between right and wrong practices:

Aspect	Best practice	Pitfall
Secrets	SOPS or external manager	Committed in plaintext
Health check	`healthChecks` in Kustomization	Left empty
Validation	CI renders and diffs before merge	Apply straight to main
Alerting	Alerts for failed conditions	No monitoring
Rollback	Nested Kustomization + pinned revision	No plan to go back

#Closing

In this episode you put together a complete FluxCD pre-production checklist: HA and RBAC, encrypted secrets, monitoring and alerting, tested backup, documented DR, plus operational practices, Git governance, and a list of pitfalls to avoid.

The key takeaways:

Prepare from the bottom up: HA, RBAC, secrets, monitoring, backup, and DR are one package, not an option.
Operations is routine: scheduled upgrades, capacity monitoring, and maintained runbooks.
The fleet is production code: branch protection, CODEOWNERS, and PR templates apply fully in the fleet repository.
Avoid the most common pitfalls: validate before merge, health checks for everything, and a clear rollback strategy.
Test the recovery: a new cluster that can rise only from Git is proof that GitOps works.

In episode 35, this series closes by looking to the future: Future of FluxCD & GitOps — the Flux roadmap, the evolution of the GitOps ecosystem, emerging patterns, platform engineering trends, and a summary of the best practices from your whole journey. See you!