Learn MicroCloud - Storage: MicroCeph

#Introduction

The cluster is formed from episodes 4-5. Now we go down to the layer that matters most for your cloud's reliability: storage. In episode 6 we dissect MicroCeph — how empty disks become OSDs, how data is replicated, and the three storage interfaces Ceph provides: block (RBD), file (CephFS), and object (RGW).

A fitting analogy: Ceph is a decentralized logistics warehouse. Goods (data) are split into small pieces (objects), spread across many warehouses (OSDs) in many cities (nodes), and each piece is copied several times (replication). If one warehouse burns down, the goods stay available from copies in other warehouses. This is why your storage can "never be lost" even if a node dies.

#Ceph Basics

#OSD: The Storage Unit

An OSD (Object Storage Daemon) is a daemon managing one disk. Each empty disk you selected during init/join becomes one OSD. An OSD stores data, replicates, and reports its health to the cluster. In MicroCloud, managing OSDs means managing disks:

microceph status

Health: HEALTH_OK
  Monitors: 3
    node-a (192.168.1.10)
    node-b (192.168.1.11)
    node-c (192.168.1.12)
  OSDs: 3
    osd.0 (node-a)
    osd.1 (node-b)
    osd.2 (node-c)

HEALTH_OK is the ideal state — it means all monitors are present (quorum) and all OSDs are active.

#Replication for HA

Ceph stores several copies of every piece of data. On a 3-node cluster, MicroCloud uses replication factor 3 by default: three copies, each on a different node. The consequences:

• One node dies → two copies remain → data stays available, the cluster stays HEALTH_OK (or WARN temporarily).
• Two nodes die → not enough copies remain to guarantee consistency → the cluster can enter HEALTH_ERR.

This is why you need at least 3 nodes for true HA. The trade-off: effective capacity = disk capacity ÷ replication factor (we calculate this in episode 18).

#Pools: RBD, CephFS, and RGW

#The RBD Block Pool

The RBD (RADOS Block Device) pool provides raw block volumes — this is the primary storage for LXD instances. Every time lxc launch creates an instance with Ceph storage, LXD creates an RBD volume for that instance's root disk.

ceph osd pool ls
ceph osd pool autoscale-status

#The CephFS File Pool

CephFS is a POSIX distributed filesystem — it can be mounted as an ordinary directory on many hosts at once. Great for:

• Sharing files between instances.
• Shared home directories for workloads that need standard paths.
• Simple backup dumps.

sudo mkdir -p /mnt/cephfs
sudo mount -t ceph :/ /mnt/cephfs -o name=admin

#The RGW Object Pool (S3)

RGW (RADOS Gateway) provides an Amazon S3-compatible API for object storage. This opens MicroCloud up to object-native workloads — for example media files, artifacts, or object backups. With standard S3 tools:

microceph rgw

s3cmd ls --host=$(microceph rgw) --no-ssl

The combination of all three makes MicroCloud more than a hypervisor — it's also a general-purpose storage provider with three interfaces from a single set of disks.

#Adding Disks: microceph disk add

#Adding an OSD

Storage doesn't have to be static. You can add a new disk as an OSD at any time:

sudo microceph disk add /dev/sdc

Verify the new OSD appears:

microceph status
ceph osd tree

#Placement Groups (PGs)

Ceph data is mapped to storage via PGs (placement groups) — the logical unit connecting objects to OSDs. The number of PGs determines the granularity of data distribution:

• Too few PGs → uneven distribution, one OSD can get overwhelmed.
• Too many PGs → memory and operational overhead increase.

MicroCloud automates PG selection when a pool is created (autoscale), and this value can be viewed/tuned:

ceph osd pool autoscale-status

#Disk Encryption (Optional)

MicroCeph supports LUKS encryption for OSD disks. When adding a disk, you can enable encryption so the data on disk is encrypted at rest:

sudo microceph disk add /dev/sdc --wipe

Encrypt this disk with LUKS? (yes/no) [default=no]

Encryption adds a layer of protection for sensitive data (compliance, used disks) at a small performance cost. We cover encryption and data protection thoroughly in episode 15.

#Common Pitfalls

• HEALTH_WARN after adding a disk: normal during data rebalancing; wait until ceph -s shows HEALTH_OK again.
• Undersized PGs: if only two OSDs exist, a replication factor of 3 can't be satisfied — MicroCloud adjusts; make sure the third node joins before loading a lot of data.
• Using --wipe on a disk containing data: --wipe intentionally formats — make sure you're pointing at a truly empty disk.
• CephFS mount fails: check that the CephFS pool exists (ceph fs ls) and the admin keyring is accessible.

#Closing

Key takeaways:

• An OSD = one disk = one daemon; MicroCeph manages OSDs via microceph status and microceph disk add.
• Replication (default 3 copies on 3 nodes) is the key to storage HA.
• Three interfaces from a single set of disks: RBD (block), CephFS (file), RGW (S3 object).
• PGs manage data distribution; autoscaling is enough for small-to-medium clusters.
• Optional LUKS encryption is available when adding an OSD.

In the next episode, we'll cover networking: MicroOVN — an SDN overlay with logical switches and routers, an optional dedicated underlay, networks for LXD instances, and the microovn status and lxc network list practices. Your cloud's network will connect automatically!