Learn OpenClaw - Disaster Recovery & Failover

#Introduction

Episode 17 closed with multi-cluster and hybrid environments: how policies are managed across many clusters, gateways at every entry point, and cross-cloud network considerations. Episode 18 goes deeper into the reliability side: disaster recovery & failover. The main reason is simple — policy is the backbone of network security. If policy configuration is lost, control over traffic is lost. If the control plane dies without a replacement, no new policy can be applied.

Episode 18's roadmap: getting to know RPO and RTO as design targets, arranging OpenClaw configuration backup and restore, designing control plane failover strategies, handling network partitions, then testing the recovery plan before it's actually needed.

#Backup and Restore of OpenClaw Configuration

What should be backed up? Imagine all the state that makes your network behave the way it does now. That includes every policy resource (ServicePolicy, NetworkPolicy, MeshConfig, ServiceIdentity), global config maps, the secrets underpinning the CA and mTLS certificates, all the way to the CRD definitions themselves. Without the CA secret, already-deployed service identities can't be verified after a restore.

Building a routine. A single backup isn't enough — schedule periodically with Velero or a CronJob, keep several versions, and mark each artifact with a checksum so a restore never uses a corrupt file. Store copies in a location separate from the primary cluster, even in a different region, so a disaster that destroys the cluster doesn't destroy its backup too. Also mind retention: keep long enough for compliance needs, but not forever.

kubectl get crd -l openclaw.io/active -o yaml > openclaw-crds.yaml
kubectl get servicepolicies,meshconfigs,serviceidentities,auditconfigs --all-namespaces -o yaml > openclaw-policies.yaml
kubectl get secrets -n openclaw -o yaml > openclaw-secrets.yaml
openclawctl backup create --name pre-release --destination s3://openclaw-backups

kubectl apply -f openclaw-crds.yaml
kubectl apply -f openclaw-secrets.yaml
kubectl apply -f openclaw-policies.yaml
openclawctl policy validate --cluster restored-cluster

The restore order matters: make sure CRDs are installed first, then secrets, then policy resources. Always test restoring from the latest backup, not just an old one — a backup never restored is only a security illusion.

#Control Plane Failover Strategies

The OpenClaw control plane must run more than one replica. When one replica dies, another takes over through leader election: one replica becomes the leader that writes, the rest serve reads and stand ready to replace it. So decisions don't split, the concept of quorum is used — a majority of replicas must agree. With three replicas, losing one node is still safe; with two, losing one destroys quorum and writes stop.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: openclaw-controlplane
  namespace: openclaw
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: openclaw-controlplane

It's not just the replica count, but where they sit. With the topologySpreadConstraints above, the three replicas are forced to spread across different zones, so the failure of one availability zone doesn't kill the whole control plane. This is the foundation of the HA architecture you touched on in episode 15.

Choose a failover strategy according to your RTO. Active-active means all replicas in all clusters serve traffic — fast failover, but high complexity. Active-standby means a standby cluster monitors along and just gets promoted when the primary collapses — simpler, but with a longer RTO because it needs manual action. You can even place the standby on a different cloud provider, in the spirit of the multi-cluster setup from episode 17.

openclawctl controlplane status
openclawctl controlplane failover --to standby-cluster

#Handling Network Partitions

A network partition happens when the cluster splits in two because the network between nodes is severed. Data planes that already received policies keep enforcing them from cache — this is the last-known-good behavior: enforcement continues, but changes can't be applied until quorum returns. This design deliberately keeps the network from collapsing entirely when the control plane is unreachable, but the consequence must be understood: policy changes during the partition are held back.

The biggest danger of a partition is split-brain — two sides of the cluster each thinking they're the leader. The prevention is the lease and quorum discussed earlier: a replica that loses contact with the majority must not accept writes, even if it thinks it's the healthiest.

apiVersion: openclaw.io/v1
kind: ControlPlaneConfig
metadata:
  name: default
spec:
  ha:
    quorum: majority
    leaseDuration: 15s
    renewDeadline: 10s
    retryPeriod: 2s

When the partition ends, don't immediately trust either side's state. Verify by comparing policies on both sides against the GitOps source of truth: openclawctl policy diff --from cluster-a --to cluster-b. If there are differences, let git win — the diverging side is reconciled again, not merged manually.

#Testing the Recovery Plan

A recovery plan that was never tested is a plan that doesn't exist. Make testing a scheduled event, not an emergency response. Every time you test, record how long recovery actually takes — that's your real RTO, not the number on a document.

kubectl delete pod -l app=openclaw-controlplane -n openclaw
watch -n 2 openclawctl controlplane status
kubectl rollout status deployment/openclaw-controlplane -n openclaw --timeout=60s

A minimum list of scenarios worth testing: full restore into an empty cluster, sudden leader death, a partition between zones, and a data plane cut off from the control plane. For each scenario, write the results into the runbook — what ran fast, what failed, what must be fixed. Recovery is a muscle: the more often it's trained, the faster it responds when needed.

#Wrap-Up

In episode 18 you understood that OpenClaw reliability is built on three pillars: tested backup, designed failover, and understood partitions. A backup without restore is nothing, failover without quorum risks split-brain, and a partition without a fail-open or fail-closed decision can hurt in both directions.

Key takeaways:

• Backups must be tested with a real restore, stored outside the cluster, and have their checksums verified.
• The control plane needs an odd number of replicas spread across zones so quorum and failover work.
• Choose between active-active and active-standby based on the RTO the business can accept.
• During a partition, rely on last-known-good and the git source of truth for reconciliation, not manual merging.
• Schedule recovery game days and measure the real RTO, not the one written in documents.

In episode 19, you enter Phase 6 — CI/CD & GitOps for Policies. You'll build a pipeline for policy changes, validate and deploy automatically, and promote policies across branches and environments with GitOps discipline. See you there!