Learn MicroCloud - High Availability & Failover

#Introduction

In episode 8 we deployed instances and VMs. But a reliable cloud isn't just able to run workloads — it must survive when a node dies. In episode 9 we test high availability: how the LXD cluster reschedules instances, how Ceph protects data, and how you verify automatic recovery through a node-down simulation.

A fitting analogy: HA is a colony of ants regenerating. If one ant (node) dies, the colony (cluster) doesn't panic — workers (instances) that lost their nest are relocated, food reserves (Ceph replication) stay stored in other nests, and the queen (quorum) keeps decisions valid. What you learn today is how this colony's "instinct" works technically.

#HA Mechanisms in MicroCloud

#Two Protection Layers

MicroCloud's HA works at two complementary layers:

1. LXD cluster — instances that are scheduled can be moved to another node when their origin node dies.
2. MicroCeph — instance data is replicated to other nodes, so the moment an instance is moved, its disk is already available on the destination node.

node-a down ──► LXD: mark offline, reschedule instance
                Ceph: salinan data tetap ada di node-b/c
                OVN: IP instance tetap valid via overlay

Without either layer, HA is incomplete: LXD could move an instance but there's no data at the destination (local storage), or Ceph is safe but instances aren't automatically rescheduled.

#Why 3 Nodes?

An LXD cluster uses quorum to make decisions. With 3 nodes:

• 1 node dies → quorum still exists (2/3) → the cluster keeps running and decides on failover.
• 2 nodes die → no quorum → the cluster stops making decisions to avoid split-brain.

Odd numbers (3, 5) are chosen for maximum failure tolerance: 3 nodes tolerate 1 failure, 5 nodes tolerate 2 failures. This is why "a cloud inside 3 nodes" is MicroCloud's magic number.

#Node-Down Simulation

#Preparing Failover-Resilient Instances

Before simulating a failure, make sure instances use Ceph storage (the remote pool) and the OVN network:

lxc list
lxc config show c1 | grep -E "pool|network"

All instances we want to test must be Ceph-based. Instances on local storage will die along with their node — that's intentional behavior.

#Taking a Node Down

Simulate a failure by shutting down one of the nodes where an instance runs. From another node or the physical host:

sudo poweroff

Or for a subtler simulation, only stop its LXD daemon:

sudo snap stop lxd

#Observing Recovery

From node-a, watch how the cluster responds:

lxc cluster list

+--------+---------+----------------------+--------+-----------+---------+
|  NAME  |  STATUS |        ADDRESS       | ROLES  | MESSAGE   | ...
+--------+---------+----------------------+--------+-----------+---------+
| node-a | ONLINE  | 192.168.1.10:8443    | leader |           |
| node-b | OFFLINE | 192.168.1.11:8443    |        | Unavailable |
| node-c | ONLINE  | 192.168.1.12:8443    |        |           |
+--------+---------+----------------------+--------+-----------+---------+

LXD marks node-b as OFFLINE and starts rescheduling movable instances:

lxc list

Instances that were LOCATION: node-b will appear with the STOPPED status and then be scheduled to node-c or node-a (for system containers; VMs with certain configs can be migrated statefully). The data stays intact because it's stored in Ceph.

#Restoring a Node

#Node Comes Back Online

When node-b comes back to life, the LXD cluster detects it and marks it ONLINE again:

lxc cluster list

Ceph also heals itself — the returning node-b OSD will participate in re-replicating any data that was lost (recovery). During this process, ceph -s shows recovery and a temporary HEALTH_WARN.

ceph -s

#Verifying No Data Was Lost

lxc list
lxc exec c1 -- df -h /

All instances must stay alive with intact data — proof that Ceph replication works.

#Recommended HA Drills

A set of drills you should run in your lab:

1. Container on Ceph storage → kill its node → observe automatic rescheduling.
2. VM → kill its node → observe the VM restarting on another node.
3. VM live migration → lxc move --target while the node is alive → observe zero-downtime.
4. Two nodes dying at once → observe that the cluster makes no decisions (no quorum) → turn on one node → the cluster recovers.

Drill number 4 is the most important to understand: losing quorum halts decisions, not destroys data. Once a node returns, the cluster recovers itself.

#Common Pitfalls

• Instances on local storage die along with the node: that's expected — for HA use the Ceph pool.
• Cluster "stuck" when a node dies: if two of three nodes die, wait for one to return; don't re-initialize midway.
• VM migration fails due to resources: the destination node must have enough CPU/RAM; check lxc info --resources per node.
• Expecting instant recovery: rescheduling and Ceph recovery take time (minutes, depending on data size) — not seconds.

#Closing

Key takeaways:

• HA = LXD reschedules instances + Ceph replicates data + OVN keeps IPs stable.
• 3 nodes give 1-failure tolerance; quorum keeps decisions valid.
• Ceph-based instances are automatically rescheduled when their node goes down.
• Ceph heals itself when a node returns (recovery).
• Always drill the node-down scenario before bringing your cloud to production.

In the next episode, we'll cover snapshots, backups & restore — lxc snapshot c1 snap1, storage pool snapshots, RBD snapshots in Ceph, lxc export/import, and backup integrations like restic and Proxmox Backup Server inside a VM. Your data will have a safety net!