Learning GitOps - FluxCD - Troubleshooting & Debugging

#Introduction

In episode 31 you arranged the enterprise adoption with mature processes and organizational structure. But no matter how tidy the configuration, in production problems will still appear — and what separates a great engineer is the speed at finding the root cause. In this episode 32 you'll learn systematic FluxCD troubleshooting & debugging: from the most common problems, to investigation techniques using the CLI and deep observation.

#Common Problems

#Reconciliation Failures

When Flux can't pull or apply configuration, all affected Kustomizations enter the Not Ready status. Start with flux get kustomizations to see the condition of each object.

#Unavailable Source

A failing GitRepository means Flux can't pull from Git. Common causes: the repository doesn't exist, the branch changed, wrong credentials, or a GitHub/GitLab rate limit.

Note

Always check the message on the object's status. Flux writes the specific reason there — for example "repository not found" or "authentication required".

#Failed Health Checks and Authentication

A health check fails when the readiness probe never becomes successful within the time limit defined in spec.healthChecks. Meanwhile, authentication and RBAC errors usually appear when Flux tries to read or write resources not allowed by the cluster's RBAC policy.

#Basic Debugging Techniques

#flux logs

The fastest way to see what Flux is doing is streaming the logs of all controllers:

Stream the logs of all Flux controllers

flux logs --all-namespaces --since=10m

Add --level=error to see only errors, or target a specific controller with --kind=Kustomization.

#flux events

Because container restarts wipe logs, Flux also writes events to the cluster as Kubernetes Events. Display them with:

View the events produced by Flux

flux events --all-namespaces --since=1h

Events are a trace that outlives the logs — very useful for problems that only appear occasionally.

#flux trace

For problems involving the source-to-kustomization flow, flux trace visually shows the relationship between objects:

Trace the origin of a resource

flux trace apps Deployment shop

#kubectl describe and Event Inspection

Combine it with the conventional way: kubectl describe kustomization shop -n apps to read the latest condition, then kubectl get events -n apps --sort-by=.lastTimestamp to see the sequence of events.

#Debugging with the Flux CLI

#flux get all

In one look, all Flux resources in the namespace:

Summary of all Flux resources

flux get all --all-namespaces

This gives a map of the overall status — from sources, kustomizations, and helm releases to image policies.

#flux reconcile and suspend/resume

To trigger an immediate reconciliation without waiting for the interval:

Force reconciliation and follow along

flux reconcile source git flux-system --with-source
flux reconcile kustomization flux-system --with-source

If a change locks up consecutive reconciliations, pause it with flux suspend kustomization shop, fix it, then continue with flux resume kustomization shop.

Caution

suspend stops reconciliation without deleting existing resources. Make sure you note which objects are suspended, because they're easy to forget and can leave the environment un-updated.

#flux export

Exporting resources as YAML is useful for inspecting the state stored in the cluster:

Export a Flux object as YAML

flux export gitrepository shop

#Advanced Debugging

#Controller Log Analysis

When the message in the status isn't enough, dig into the controller logs directly:

Log of a specific controller with filters

kubectl logs -n flux-system deploy/source-controller \
  --follow --tail=200

#Metrics Inspection

Flux controllers expose Prometheus metrics, including gotk_reconcile_condition and gotk_reconcile_duration. A False value on gotk_reconcile_condition indicates a problem worth investigating:

Example of a Prometheus query result

{
  "metric": {
    "__name__": "gotk_reconcile_condition",
    "kind": "Kustomization",
    "name": "flux-system",
    "status": "False"
  },
  "value": [1722657600, "1"]
}

#Network and Git Debugging

Network issues often appear as timeout or TLS errors. Use kubectl exec into the controller pod and try git ls-remote directly to isolate whether the problem is in the connection or the credentials:

Test the git connection from inside the pod

kubectl exec -n flux-system deploy/source-controller -- \
  git ls-remote https://github.com/devvnull/fleet.git

Tip

Build a fixed investigation sequence: check flux get all, then events, then logs. This order prevents you from jumping straight to a wrong assumption.

#Closing

In this episode you learned systematic FluxCD troubleshooting: recognizing the five common problems, reading logs and events, using flux trace, leveraging flux reconcile and suspend/resume, and digging into controller logs and metrics.

The key takeaways:

Status first, assumptions later: read the object conditions and events before concluding the cause.
Logs are a temporary narrative, events are the record: use flux events for problems that rarely appear.
Master the CLI: flux get all, flux reconcile, and flux suspend/resume handle most fixes.
Isolate the layers: separate Git, network, and RBAC problems by testing one by one.
Export for analysis: flux export and Prometheus metrics speed up diagnosis.

In episode 33, we discuss a journey that's no less challenging: migration strategies — moving from manual deployments, Helm 2, other GitOps tools, and especially the migration from Flux v1 to v2 and from ArgoCD to Flux. See you!