Learning GitOps - FluxCD - Sharding & Performance

#Introduction

In episode 23 you learned multi-cluster management — managing many clusters with the hub-and-spoke pattern and configuration distribution. But the bigger the cluster and the more resources Flux watches, the next question arises: are all the reconciliations still running smoothly?

The problem isn't just adding CPU. At a certain scale, a single Flux instance starts to struggle: hundreds of Kustomizations, thousands of resources, and many GitRepositories refetched on the same interval. This isn't a sign that Flux can't scale — it's a sign the architecture needs to be adjusted, just like an application architecture.

In this episode we discuss how to keep Flux fast and reliable at scale: horizontal sharding, vertical sharding, reconciliation performance optimization, and large-scale repository patterns.

#Horizontal Sharding

#Dividing the Load with Multiple Flux Instances

Horizontal sharding in Flux is the same as database sharding: instead of one instance handling all resources, the load is divided among several instances, each handling a subset. The advantages: each instance has its own resources, they don't fight over CPU and memory, and one shard's failure doesn't take everything down. Sharding is enabled from bootstrap with the --sharding and --shard flags:

flux bootstrap github \
  --owner=devvnull --repository=gitops-production \
  --path=clusters/production \
  --sharding --shard=0/2

flux bootstrap github \
  --owner=devvnull --repository=gitops-production \
  --path=clusters/production \
  --sharding --shard=1/2

The second shard is declared 1/2 — the second part of a total of two parts, numbered from zero.

#Sharding by Namespace

How sharding works in Flux: every resource is given a label derived from the hash of the namespace where the resource lives. The result is deterministic — the same namespace is always handled by the same shard, so no two controllers fight over a resource. Check the distribution with the label:

kubectl get kustomization -A -l sharding.fluxcd.io/key
kubectl get pod -n flux-system -l app=kustomize-controller

The sharding.fluxcd.io/key label determines the owning shard, with the value being the namespace's numeric hash modulo the total number of shards. This pattern can be combined with namespace per tenant: each tenant falls into its own shard, giving load isolation — one tenant doing large deployments doesn't slow down another tenant's reconciliation.

#Even Load Distribution

Hash-based distribution isn't a mathematical guarantee of evenness. Monitor it periodically:

flux stats
kubectl top pods -n flux-system

flux stats gives a picture of the reconciliation count; kubectl top pods shows CPU and memory usage per controller. If one shard is consistently higher, increase the number of shards — for example from 2 to 4 with --shard=0/4 through --shard=3/4.

#Vertical Sharding

Vertical sharding divides the load by type of responsibility, not quantity. One cluster can run several isolated Flux instances in their own namespaces. The most common split is infrastructure vs applications: the infrastructure Flux manages operators, ingress, cert-manager, and monitoring (rarely changes, big impact), while the application Flux manages business deployments that change often and need fast reconciliation. A second split is core vs edge: resources in the central datacenter versus resources at the edge locations, where the edge often needs sources that can be reconciled offline — for example replacing GitRepository with an OCIRepository that's faster and cheaper.

#Reconciliation Performance Optimization

#Tuning the Reconciliation Interval

A uniform, very small interval for all resources is the biggest waste in a large cluster. A wise strategy: rarely changing infrastructure uses interval: 1h or more; active applications 5m to 15m; only critical things 1m.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 5m
  url: https://github.com/devvnull/gitops-apps
  ref:
    branch: main

Also pay attention to the timeout on the Kustomization — the rule of thumb is at least twice the longest reconciliation duration you've ever measured.

#Resource Limits and Concurrency

Without limits, controllers can consume cluster resources without bounds; with limits that are too tight, they get oom-killed often. Start from conservative values, then raise them based on kubectl top observation:

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 1Gi

In addition, the --concurrent flag on the controller limits parallel reconciliation. The default of 4 is enough for most cases; raise it only if the controller is idle while the queue piles up.

#Caching and Garbage Collection

Flux caches at several layers: source artifacts (fetch results aren't repeated if nothing changed), HTTP cache to the registry and git hosts, and discovery cache for health assessment. Avoid changing spec.ref too often — every change invalidates the cache. Check when a source was last fetched with flux get sources git -A.

Garbage collection (pruning) removes resources that exist in the cluster but not in Git. It must be enabled through spec.prune:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  sourceRef:
    kind: GitRepository
    name: apps
  path: ./apps
  prune: true
  interval: 5m

#Large-Scale Patterns

#Monorepo vs Polyrepo

A combination often used: a monorepo for the platform (shared infrastructure) and polyrepos for applications (each team holds its own repo). A recommended repo structure looks roughly like:

clusters/
  production/
    flux-system/
    infra/
    apps/
apps/
  checkout/overlays/production/
  payment/overlays/production/
platform/
  nginx-ingress/
  cert-manager/
  monitoring/

#Git and Webhook Performance

Keep the repo lean: don't commit artifacts (images are just referenced by tag), use stable refs, and consider an OCIRepository for the best performance. Webhooks speed up detection without waiting for the interval, through a Receiver:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: github-receiver
  namespace: flux-system
spec:
  type: github
  events:
    - "push"
  secretRef:
    name: webhook-token
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      name: apps

#Closing

In this episode 24 you brought Flux to production scale: horizontal sharding to divide the load by namespace, vertical sharding to separate responsibility domains, performance optimization through intervals, resource limits, caching, and pruning, and large-scale repository patterns with an efficient structure and webhook optimization.

The key takeaways:

• Horizontal sharding divides the load across several instances with namespace-hash-based labels — measure the load first before deciding the number of shards.
• Vertical sharding separates domains so failures and policies don't spread to each other.
• Intervals are tuned per resource, not a uniform minimum — combine them with webhooks for latency and efficiency.
• Resource limits, concurrency, and caching are the main performance levers; watch them with kubectl top.
• Pruning must be enabled so the cluster doesn't pile up orphaned resources.

Your clusters are now fast and stable. But how do we know everything is healthy? In the next episode, episode 25, we'll discuss Monitoring & Observability — Prometheus metrics, Grafana dashboards, log aggregation, distributed tracing with OpenTelemetry, and alerting for failed reconciliation. Keep up the momentum!