Learn MicroCloud - Troubleshooting

#Introduction

After encryption and data protection in episode 15, you have a secure cloud. But security doesn't mean problem-free — nodes can fail to join, OSDs can drop, and clusters can become degraded. In episode 16 we build your troubleshooting toolkit: how to read status, dig into logs, solve common cases, and recover the cluster when something breaks.

A fitting analogy: pilots don't wait for a plane to crash to learn the emergency checklist. They memorize procedures before problems arise, so when a red indicator lights up, their hands already know what to do. This episode is your emergency checklist for MicroCloud — save it, understand it, and use it when the cluster misbehaves.

#Debug Toolkit: Status and Logs

#microcloud status

The first command you should remember is microcloud status — it summarizes the health of the whole stack (LXD, Ceph, OVN) on one screen:

sudo microcloud status

Clustering                   |  enabled
                           ...
MicroCloud services          |  MicroCloud, MicroCeph, MicroOVN, LXD
                            |  all running
LXD cluster                  |  3 members, all online
Ceph cluster                 |  HEALTH_OK
OVN cluster                  |  all online

If one layer has a problem, microcloud status usually shows it here — this is the always starting point before diving into logs.

#snap logs microcloud

microcloud status tells you what's wrong; logs tell you why. Every snap stores logs readable with snap logs:

sudo snap logs microcloud -n 200

#microceph status and microovn status

For storage problems, dig deeper into MicroCeph; for networking, into MicroOVN:

sudo microceph status
ceph -s

sudo microovn status
sudo snap logs microovn -n 200

#LXD and Instance Logs

LXD records cluster and instance activity. If an instance fails to start or the cluster acts "weird", check the LXD logs:

sudo snap logs lxd -n 200

#Common Cases and Solutions

#Failed Join

A failed join is the most common complaint. Symptom: microcloud join on a new node stops with a token or timeout message. The most frequent causes:

• Expired token: since 2.1.3, join tokens are created with a default validity of 1 hour (previously 5 minutes). If the join takes long, create a new token from the first node.
• Duplicate hostname: a node with an already-used hostname will be rejected — make sure the hostname is unique (hostnamectl hostname).
• Nodes can't reach each other: verify connectivity and open ports.

hostnamectl hostname
ping -c 3 node-b
sudo microcloud status

#Small Disks

Symptom: an OSD is active but the cluster is HEALTH_WARN with near full, or data can't spread evenly. Check usage and see if any OSD is much smaller than its siblings:

lsblk -d -o NAME,SIZE,TYPE
ceph osd df
ceph osd tree

If one OSD is FULL while others are comfortable, data is unbalanced. The short-term fix is letting Ceph rebalancing finish; the long-term fix is adding disks (episode 11) or reducing replication if capacity is genuinely limited.

#Underlay Network

Symptom: the cluster is healthy for a while, then members "disappear", or storage traffic is slow. Underlay problems — the physical network where LXD/Ceph/OVN communicate — are often the culprit:

ping -c 5 node-a
mtr -n node-b

Also check that inter-node ports aren't blocked by a firewall (episode 14 covers the segments and key ports: 8443 LXD, 6789/6800-7300 Ceph, 6081 OVN tunnels).

#Clock Skew

LXD and Ceph are very sensitive to time differences between nodes. If a member is suddenly marked offline while the network is healthy, suspect the clock:

timedatectl status
chronyc sources -v

All nodes must use the same time source (e.g., chrony) — a large offset will make LXD certificates and heartbeats invalid.

#Member Recovery

#Removing a Member Properly

If a node is permanently broken and must leave the cluster, don't remove it from LXD only — MicroCloud 2.x provides a dedicated command that cleans up across all services at once:

sudo microcloud remove node-b

Run it from another healthy node, then verify:

sudo microcloud status
lxc cluster list

#Maintenance Without Removal

For planned maintenance — reboots, hardware swaps — you don't need to remove a member. Use evacuate to move all instances off first, then restore when the node returns:

lxc cluster evacuate node-b
# ... maintenance ...
lxc cluster restore node-b

#Service Re-init

If a snap's state is corrupted and unrecoverable, the last resort is re-init: purge all snaps and reinstall. This is a nuclear step — only do it when truly stuck:

sudo snap remove microcloud --purge
sudo snap remove microovn --purge
sudo snap remove microceph --purge
sudo snap remove lxd --purge

Once clean, reinstall (episode 3) and microcloud join back into the cluster. Proactive trust (episode 15) helps a re-init'd node rejoin without friction.

#Common Pitfalls

• Removing a member from only one service: leaves leftover state in LXD/Ceph/OVN — always use microcloud remove.
• Rebooting without evacuating: instances on local storage can be lost; Ceph instances get rescheduled but with downtime — evacuate first.
• Ignoring logs: a healthy microcloud status doesn't guarantee clean logs — read the logs for intermittent issues.
• Re-init too quickly: re-init wipes node config data; make sure backups (episode 10) are safe before the nuclear step.
• Guessing without data: data-driven diagnosis (status + logs + network checks) is always faster than guessing.

#Closing

Key takeaways:

• Start debugging from microcloud status, then dig into the relevant snap logs.
• Failed join: check the token, unique hostname, and inter-node connectivity.
• Small disks and unbalanced data: monitor ceph osd df and let rebalancing finish.
• Intermittent "missing" nodes: suspect the underlay network and clock skew before guessing software.
• Recover members with sudo microcloud remove <node>; maintenance uses lxc cluster evacuate/restore.
• Re-init a service only as a last resort, and only with backups already safe.

In the next episode, we'll cover MicroCloud 3.2 & the latest features — LXD 6.8 compatibility, fixes and features in the 3.2 feature release (April 2026), why 3.x isn't yet recommended for upgrading from 2 LTS, and the 2.1.3 LTS release for production with an official upgrade path. Version matters!