Learn MicroCloud - Scaling: Adding Nodes & Disks

#Introduction

So far you've managed a static cloud — three nodes, three OSDs, fixed capacity. But workload needs always grow: more instances, more data. In episode 11 we learn how to scale MicroCloud in two dimensions: adding nodes (new compute + storage) and adding disks (per-node capacity). Both can be done without stopping the cluster.

An analogy: a cluster is like a coalition of companies. Adding a node is like bringing in a new partner — they bring their own capital (OSD disks) and share the workload. Adding a disk is like a partner buying a new warehouse — storage capacity rises, and goods (data) are automatically redistributed to stay balanced. Nobody has to "change offices".

#Adding a New Node

#Preparing the New Node

The fourth node needs the same prerequisites as episode 3:

sudo hostnamectl set-hostname node-d
sudo apt install -y chrony && sudo systemctl enable --now chrony
sudo snap install lxd microceph microovn microcloud

#Joining a Live Cluster

Unlike episode 5 (joining at the same time as init), here the cluster is already running. The new node discovers the cluster via mDNS and joins:

sudo microcloud join

MicroCloud scans the network, finds the existing cluster, and asks the usual storage/network questions. After peer approval (manual or proactive trust), the new node becomes a member:

lxc cluster list
microcloud status

+---------+---------+-------------------+------+------+
|  NAME   | STATUS  |      ADDRESS      | ...  |      |
+---------+---------+-------------------+------+------+
| node-a  | ONLINE  | 192.168.1.10:8443 |      |      |
| node-b  | ONLINE  | 192.168.1.11:8443 |      |      |
| node-c  | ONLINE  | 192.168.1.12:8443 |      |      |
| node-d  | ONLINE  | 192.168.1.13:8443 |      |      |
+---------+---------+-------------------+------+------+

#New Quorum and Tolerance

With 4 nodes, the failure tolerance stays at 1 (because an absolute majority, 3 out of 4, is required). Tolerance rises to 2 once the cluster has 5 nodes. That's why adding nodes in odd numbers (3 → 5) gives a real tolerance jump.

#Automatic Ceph Rebalancing

#What Happens When a New Node Arrives

When the new node's OSD becomes active, Ceph starts rebalancing: some PGs (placement groups) are moved to the new OSD so capacity and load are spread evenly across the cluster. This happens automatically — no manual commands needed.

ceph -s

cluster:
  id:     ...
  health: HEALTH_WARN
          Rebalancing after adding osd.3

#Controlling the Speed

Rebalancing can strain the network. To be gentle, limit the recovery rate:

sudo microceph cluster config set osd_max_backfills 1
sudo microceph cluster config set osd_recovery_max_active 1

After rebalancing completes (back to HEALTH_OK), restore the defaults:

sudo microceph cluster config set osd_max_backfills 4
sudo microceph cluster config set osd_recovery_max_active 5

#Adding OSD Disks

#Adding an OSD on an Existing Node

Capacity can also grow by adding disks to nodes that are already members:

sudo microceph disk add /dev/sdc

microceph status
ceph osd tree

#Expanding the LXD Storage Pool

Once Ceph capacity grows, the LXD pool automatically benefits from the new capacity — no pool reconfiguration needed. You can also create additional pools with different policies (e.g., replication 2 for less critical data):

sudo microceph pool add pool-fast replication 2

lxc storage create fast ceph --source pool-fast

Now there's a new LXD storage pool named fast using the Ceph pool pool-fast with replication 2 — saving capacity for data that can tolerate higher risk.

#Realistic Scaling Scenarios

• Full capacity → add an OSD (microceph disk add) — fast, no new member needed.
• Need more compute → join a new node (microcloud join) — capacity rises in every dimension.
• Want higher tolerance → make sure the total node count is odd (3 → 5, not 4).
• Special pool with lower replication → microceph pool add + lxc storage create.

#Common Pitfalls

• Slow rebalancing: with large data and a slow network, rebalancing can take hours — limit osd_max_backfills so production isn't disturbed.
• Adding an even node for HA: 4 nodes don't increase quorum tolerance — understand the numbers before buying hardware.
• Disk too small: an OSD with a much smaller disk than the others makes distribution uneven; aim for uniform sizes.
• Forgetting time sync on the new node: clock skew on node-d is immediately noticed by Ceph — verify chrony before joining.

#Closing

Key takeaways:

• Add nodes with microcloud join to an active cluster — compute + storage capacity rises.
• Ceph rebalancing runs automatically; control its pace with osd_max_backfills.
• Add OSDs with microceph disk add — capacity grows without new members.
• New Ceph pools can be registered with LXD as additional storage pools.
• Target an odd number of nodes for a real failure-tolerance jump.

In the next episode, we'll cover multi-cluster: MicroCloud Cluster Manager — a beta tool (AGPLv3, Go, Juju + PostgreSQL + Traefik) for managing and monitoring many MicroClouds from a single UI, integrated since MicroCloud 3.1, complete with cross-cluster observability and the LXD Grafana dashboard. Your cloud is no longer alone!