Learning GitOps - FluxCD - Monitoring & Observability

#Introduction

In episode 24 you learned how to make Flux fast and reliable at scale — horizontal sharding, vertical sharding, interval tuning, and efficient repository patterns. The cluster now runs smoothly, but there's a deeper question: how do we know it's actually healthy? The honest answer: we won't know until an incident happens, unless we have good observability.

Note that performance and observability are two sides of the same coin. Without metrics, yesterday's optimizations are just guesses — we don't know which controller is the bottleneck, when a reconciliation stretches out, or whether sharding actually helps. Monitoring turns all those decisions from feeling into data.

In this episode we'll build a thorough observability foundation for Flux: the Prometheus metrics every controller exposes, Grafana dashboards both official and custom, logging and log aggregation, distributed tracing with OpenTelemetry to trace a reconciliation, and alerting so problems are known before users complain.

#Flux Metrics with Prometheus

#Controllers Provide Metrics

Every Flux controller — source, kustomize, helm, and notification — exposes a Prometheus metrics endpoint on port 8080 at the /metrics path. This makes it easy for Prometheus to scrape metrics directly from the controller Pods, even without an additional exporter. The Flux bootstrap configuration already includes scrape annotations so Prometheus can discover them automatically:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

With kube-prometheus-stack or a Prometheus Operator that follows ServiceMonitors, these annotations are enough to start scraping. For a manual Prometheus installation, add a scrape job targeting kustomize-controller:8080, source-controller:8080, and so on.

#Reconciliation and Status Metrics

The most important metrics to monitor are the ones related to the reconciliation cycle. A few of them:

• gotk_reconcile_condition — the last condition status per resource (Ready, Available), labeled with type, status, reason, kind, and namespace. This is the core metric for knowing whether reconciliation succeeded.
• gotk_reconcile_duration_seconds — a histogram of each reconciliation's duration. The initial trigger for bottleneck investigation.
• gotk_apply_duration_seconds — a histogram of the apply operation duration to the Kubernetes API.
• gotk_source_info — version information of the last source artifact (revision, commit hash).
• gotk_suspend_status — marks resources that are suspended; accidental suspensions often show up here.

To check the raw metrics directly from a controller:

kubectl -n flux-system port-forward deploy/kustomize-controller 8080
curl -s localhost:8080/metrics | grep gotk_reconcile_condition | head

gotk_reconcile_condition with status="False" means the last reconciliation failed — an alarm that must be investigated immediately. In the latest Flux versions, this metric is reported as a current-status gauge, so it can be used directly in alert rules.

#Resource Status Metrics

Besides the reconciliation metrics, there are also metrics describing the status of the resources themselves. gotk_resource_info records information about the reconciled resource, including the final health status. The combination of reconciliation metrics and resource status metrics gives a complete picture: not just whether the process ran, but also whether the final result is truly Ready.

The metric classes to distinguish when reading a dashboard:

#Grafana Dashboards

#The Official Flux Dashboard

Grafana is Prometheus's natural partner, and Flux provides an official dashboard in its repository. This dashboard can be imported directly into Grafana and already contains the important panels: reconciliation rate, duration, failures, and status per controller. To install it, add the dashboard to Grafana provisioning or import it manually through the import menu with the JSON downloaded from the Flux repository.

The panels in the official dashboard include:

• Reconciliation rate per controller — how often each controller reconciles.
• Reconciliation duration — 50th and 99th percentile histograms, to see slowdowns.
• Failed reconciliations — the count and reasons for failures.
• Source status — when a source was last successfully fetched.

#Custom Dashboards

The official dashboard is a good starting point, but every organization has unique needs. Custom dashboards are built with PromQL queries targeting specific metrics. An example of a simple panel showing Kustomizations that aren't ready:

- title: Kustomization not ready
  targets:
    - expr: |
        gotk_reconcile_condition{kind="Kustomization", type="Ready", status="False"}
      legendFormat: "{{kind}}/{{name}}"
  gridPos:
    h: 8
    w: 12

The PromQL query above shows every Kustomization whose Ready condition is False — at a glance, the team knows immediately which ones are problematic. Add similar panels for GitRepository, HelmRelease, and Receiver.

#Key Metrics to Monitor

Some metrics deserve a permanent place on the dashboard because they're always relevant:

• Reconciliation failure ratio — the percentage of failed reconciliations out of the total; a sudden spike is an early sign of a problem.
• p95 reconciliation duration — don't wait for the average; a long tail is a bottleneck indicator.
• Last source fetch time — a source that's never refetched means a webhook or interval problem.
• Suspend status — accidentally suspended resources are a classic source of confusion.

#Logging

#Controller Logs and Events

Metrics tell you what happened; logs tell you why. Every Flux controller writes logs to its Pod's stdout and can be read with ordinary commands. Flux also provides a CLI to make it easier:

flux logs --level=debug
flux logs --all-namespaces --tail=50
kubectl -n flux-system logs deploy/kustomize-controller --tail=50

Besides the controller logs, Kubernetes Events are an important source of information. When a reconciliation fails, Flux writes an event with a detailed error message:

kubectl get events -n flux-system --sort-by=.lastTimestamp

These events are often the first clue when debugging: the failure reason, the resources involved, and the time it happened.

#Log Levels

Flux supports several log levels: error, info, and debug. The debug level shows step-by-step reconciliation details — very useful when debugging, but very noisy in production. Set the level via the --log-level flag on the controller or at bootstrap:

flux bootstrap github \
  --owner=devvnull \
  --repository=gitops-production \
  --branch=main \
  --log-level=debug

Use info as the production default, and raise it to debug only during an investigation. The kustomize controller specifically requires --log-level=debug to see the diff details it applies.

#Log Aggregation with Loki or Elasticsearch

In a cluster with many controller Pods, reading logs per Pod is impractical. This is where log aggregation comes in: collect all the logs into one searchable place. Two common approaches:

• Loki + Grafana — logs are stored in Loki, searched from Grafana with labels (for example namespace and controller name). Lightweight and integrated with the dashboards that already exist.
• Elasticsearch + Kibana — full search features and a heavier load; suitable if the organization already uses an ELK stack.

With aggregation, cross-controller debugging becomes easy: trace one reconciliation from the source-controller logs to the kustomize-controller logs without hopping between Pods.

#Distributed Tracing with OpenTelemetry

#Enabling Tracing

Metrics and logs don't yet explain how time is spent inside a single reconciliation. Distributed tracing solves this problem: each reconciliation produces a trace containing spans (the steps) — fetching the source, building the manifests, applying to Kubernetes, and waiting for health. Flux supports the OpenTelemetry Protocol (OTLP) and can export traces to backends like Tempo, Jaeger, or Grafana Cloud.

Tracing is enabled per controller with the --enable-otlp=true flag and the OTEL_EXPORTER_OTLP_ENDPOINT environment variable:

containers:
  - name: manager
    args:
      - "--enable-otlp=true"
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: "http://tempo.monitoring.svc:4317"

Once tracing is active, every reconciliation produces a new trace that can be searched by resource name and revision.

#Tracing the Reconciliation Cycle

With traces, questions that were previously hard to answer become easy. Look at one Kustomization reconciliation trace: how long did the fetch take? How long did the manifest render take? How long did the apply operation take? At which point is the trace slowest?

Bottleneck patterns that often appear:

• Slow fetch — a large Git source or poor network to the git host; the solution is an OCIRepository or a closer source.
• Slow apply — the API server or admission webhooks slow down the apply; check for blocking ValidatingWebhooks.
• Slow health wait — the health assessment waits too long for a resource to become Ready; check the application's readiness.

#Alerting

#Flux's Built-in Alerting: Provider and Alert

Flux has a built-in alerting system through two CRDs: Provider (the notification destination definition, e.g. Slack, GitHub, or a generic webhook) and Alert (the rules for when and what to send). An example of sending a Slack notification on a reconciliation error:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: gitops-alerts
  secretRef:
    name: slack-webhook

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Alert
metadata:
  name: reconciliation-failed
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: '*'

eventSources with name: '*' means every Kustomization in the flux-system namespace is monitored. Every event with error severity is forwarded to Slack immediately — instant detection without waiting for the monitoring interval.

#AlertManager and Prometheus Rules

For more complex alerts — for example based on trends or periods — use Prometheus AlertManager with rules that use Flux metrics. An example rule to detect a reconciliation that has failed for more than five minutes:

groups:
  - name: flux.rules
    rules:
      - alert: ReconciliationFailed
        expr: |
          gotk_reconcile_condition{type="Ready", status="False"} == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Reconciliation failed for {{ $labels.name }}"

Combine AlertManager with rules for unavailable sources (the source controller failing to fetch repeatedly) and failed health checks (a resource not becoming Ready within the time limit). Dividing alerts by severity — warning for minor disruptions, critical for failures that impact users — keeps the notifications useful.

#Closing

In this episode 25 you built the observability foundation for Flux: Prometheus metrics from every controller with a focus on gotk_reconcile_condition and reconciliation duration, Grafana dashboards both official and custom, logging with levels and aggregation to Loki or Elasticsearch, distributed tracing with OpenTelemetry to find bottlenecks inside a single reconciliation, and alerting through Flux's built-in notifications and AlertManager rules.

The key takeaways:

• Metrics are Flux's health language — learn gotk_reconcile_condition, gotk_reconcile_duration_seconds, and gotk_apply_duration_seconds before anything else.
• The official Flux dashboard is a good starting point, then expand with custom panels that answer your organization's specific questions.
• Logs answer why, metrics answer what — combine both, and log aggregation makes cross-controller debugging practical.
• Tracing with OpenTelemetry opens up the inside of a reconciliation — enable it gradually to find bottlenecks without excessive overhead.
• Good alerts are few and precise — a combination of Flux's built-in alerts and AlertManager rules with structured severity.

Now you can see everything. The next question: what happens when everything collapses? In the next episode, episode 26, we'll discuss Disaster Recovery & Backup — backup strategies with Git and Velero, etcd snapshots, exporting Flux configuration, cluster rebuild and re-bootstrap procedures, and DR drills with measurable RTO and RPO targets. Because even the best observation means nothing without a recovery plan. Keep up the momentum!