Learn GitOps with ArgoCD - High Availability Setup

#Introduction

In episode 25 we optimized ArgoCD's performance — making it fast with thousands of applications. But speed doesn't guarantee availability: if the node hosting argocd-server dies, no matter how fast it is, the whole team loses access to the UI and CLI. In this episode we discuss high availability setup — designing ArgoCD so that a pod, node, or zone failure doesn't stop delivery.

Why does this matter? Remember ArgoCD's position: it's the only bridge between Git and the cluster. If ArgoCD dies, applications don't immediately die (Kubernetes keeps running what's already there), but everything else stops: sync, drift detection, self-healing, and operational access. For environments promising SLOs, ArgoCD itself must be the component that's hardest to kill. This episode gives the recipe — and how to prove it.

#HA Architecture

The basic HA principle is no single component is a point of failure. This is realized through three mechanisms:

• Multiple replicas — more than one pod for each stateless component.
• Leader election — for stateful components like the controller: only one is "active", the others wait; when the active one dies, its replacement takes over quickly.
• Load distribution — traffic spread across all replicas via a Service.

When this design is right, one dead pod is an ordinary event, not an incident. You'll see leader lease logs on the controller:

argocd-application-controller-0 ... starting leader election
argocd-application-controller-1 ... attempting to acquire leadership
argocd-application-controller-1 ... successfully acquired lease argocd/argocd-application-controller

#HA per Component

Let's break it down one by one.

#API Server — Multiple Replicas

argocd-server is stateless; run several replicas behind a Service:

Server HA

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: argocd-server
          args:
            - /usr/local/bin/argocd-server
            - --staticassets=/shared/app

ArgoCD handles this easily; with topologySpreadConstraints (below) replicas are spread across nodes so one node failure doesn't kill all replicas.

#Repo Server — Multiple Replicas

argocd-repo-server is also stateless, but it stores caches in Redis (not locally). So horizontal scaling is safe. Use HPA (episode 25) or fixed replicas, and make sure all replicas share the same Redis so the cache isn't fragmented.

#Application Controller — Active-Passive

The controller is stateful: it holds application state and must be executed by one leader at a time. ArgoCD solves this with leader election based on coordination.k8s.io/Lease. Run 2-3 replicas; only the leader works, the rest are standby ready to take over:

Active-passive controller

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: argocd-application-controller
spec:
  replicas: 2

When the leader dies, the lease expires and a standby takes over within seconds. This is the same active-passive pattern as the multi-cluster DR concept in episode 21, but within one cluster.

#Redis — Sentinel or Cluster

Redis is ArgoCD's cache brain. If it dies, ArgoCD still works but without a cache — performance returns to "full clone" mode. For HA:

• Redis Sentinel — one primary + several replicas with sentinels that automatically promote a replica when the primary dies. The HA chart's default choice.
• Redis Cluster — sharded, for very large scale.

Both need a PersistentVolumeClaim so data isn't lost on pod restart, and both are managed as manifests in Git:

redis:
  enabled: true
  sentinel:
    enabled: true
    masterName: argocd
  metrics:
    enabled: true

#Database Considerations

The Redis cache isn't a database that "must always be consistent" — losing it only degrades performance. But two practices are still mandatory:

• Persistent storage — give Redis a PVC so the cache survives pod restarts. Lost data isn't a disaster, but causing it means re-cloning and re-rendering all repos — a load that can be avoided.
• Backup — back up Redis (or simply accept the cache being refilled from Git). Because the source of truth is Git, the cheapest strategy is not backing up the cache, but ensuring refill runs automatically and quickly. Record this as a design decision in the documentation.

For the controller state, remember: ArgoCD's real state is already in Git. This is the structural advantage of GitOps — ArgoCD doesn't store truth, it stores results. Losing internal status only means re-reconcile, not data loss.

#Network HA

The last layer is access. The UI/CLI must stay reachable when individual components change:

• Load balancer — put a Service in front of argocd-server (NodePort/LoadBalancer/Ingress). Health checks make sure traffic only goes to healthy replicas.
• Ingress redundancy — run more than one Ingress controller; if one node dies, the controller on another node serves. Combine with the provider's network load balancer.
• DNS failover — for multi-cluster (episode 21), put a DNS record in front of a global load balancer that can redirect traffic to a secondary cluster when the primary is unhealthy.

Topology spread is an important refinement: spread replicas across different nodes so one node going down doesn't take down all replicas:

Topology spread for the server

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: argocd-server

#Testing HA: Chaos and Failure Injection

An HA architecture that isn't tested is the same as none. Use chaos engineering to prove the claims:

1. Failure injection — kill a pod at random: kubectl delete pod -n argocd argocd-server-xxx and observe whether the Service hides the replacement.
2. Kill the controller leader — make sure a standby takes over without stopping reconciliation.
3. Kill the Redis primary — make sure Sentinel promotes a replica within an acceptable time.
4. Kill one node (drain) — make sure topologySpreadConstraints works and replicas re-spread.

Tools like kube-monkey or Litmus execute these scenarios on a schedule, but even a manual monthly drill (the same pattern as the DR drill in episode 21) is hugely valuable:

kubectl delete pod -n argocd -l app.kubernetes.io/name=argocd-server --wait=false
kubectl get pods -n argocd -w
argocd app list

Recovery validation — measure the time from failure injection until all components are healthy again. Record the results: if recovery takes longer than your SLO, fix the design before a real failure happens.

#Closing

This episode closed ArgoCD's availability loop: HA architecture with multiple replicas and leader election, per-component HA (server, repo server, active-passive controller, Redis Sentinel), storage and Redis backup considerations, network HA with load balancers, ingress redundancy, and topology spread, and testing with chaos engineering and failure injection.

The points you should take with you:

• Stateless components scale horizontally; the controller uses leader election.
• Redis needs Sentinel (or cluster) and a PVC so the cache isn't a point of failure.
• ArgoCD's real state is in Git; losing internal status is only a re-reconcile.
• Topology spread distributes replicas across nodes to handle node failures.
• HA must be tested: inject failures, measure recovery, fix if it misses the SLO.

With this episode, the journey from GitOps basics to production readiness is a complete line: installation, applications, scale, security, and resilience. In the next episode 27 we discuss troubleshooting & debugging — solving sync failures, failed health checks, authentication issues, repo access, and debugging techniques with the ArgoCD CLI. See you in episode 27!