Learn Ceph - Installation & Basic Cluster Setup

#Introduction

Welcome to episode 3 of the Learn Ceph series! The first two episodes built your conceptual foundation. Now it's time to get hands-on: you'll build a real three-node Ceph cluster from scratch, complete with MON, OSD, and MGR all running normally.

There are two approaches to installing Ceph: manual install with RPM/DEB packages and cephadm, which runs daemons as containers. We'll focus on cephadm because it's the recommended method for all modern releases, including the rolling upgrades we'll cover later in episode 19.

By the end of this episode, you'll have a cluster with HEALTH_OK status, three registered nodes, and OSDs storing data. Let's get started.

#Preparation Before Installation

#Node Prerequisites

All nodes must be able to reach each other via SSH. Create the same user on every node, for example ceph, with passwordless sudo. Add the first node's public key to authorized_keys on all nodes so cephadm can control remote hosts.

useradd -m -s /bin/bash ceph
echo "ceph ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/ceph
mkdir -p /home/ceph/.ssh

Verify that from the first node you can SSH to the other nodes without a password:

ssh node2 hostname
ssh node3 hostname

The ssh node2 hostname command should return node2's hostname without asking for a password. If it still asks for a password, check the permissions of the authorized_keys file and the sshd configuration.

#Bootstrap the Cluster with cephadm

#Run Bootstrap on the First Node

Bootstrap creates the first MON, the first MGR, and the cluster keyring. Run it pointing to the mon IP:

cephadm bootstrap --mon-ip 192.168.100.10 \
  --allow-fqdn-hostname

This process will start the MON and MGR daemons, then print the dashboard URL and initial credentials to the console. Save that output — you'll need the randomly generated dashboard password.

#Deploy Additional MONs and MGRs

For redundancy, add a second and third MON, then deploy a standby MGR. Modern Ceph deploys daemons through the orchestrator:

ceph orch host add node2
ceph orch host add node3
ceph orch apply mon node1,node2,node3
ceph orch apply mgr node1,node2,node3

After these commands, the cluster has three MONs forming a quorum and an active-standby MGR. Check with ceph quorum_status to make sure the quorum is complete.

#Adding OSDs

#Deploy All Available Disks

Before adding OSDs, make sure the target disks are really empty. Run OSD deployment for all available devices:

ceph orch apply osd --all-available-devices

This command formats and initializes every empty disk it finds. Verify the results:

ceph osd tree
ceph osd stat

ceph osd tree shows the list of OSDs, the hosts they live on, and their up/down status. The number of OSDs will match the number of empty disks available across all nodes.

#Managing Individual OSDs

Sometimes you want to add OSDs one by one or point to a specific disk:

ceph orch daemon add osd node2:/dev/sdb
ceph osd status

Adding an OSD with the node2:/dev/sdb syntax gives you full control over which disk is used. This is useful when certain disks must be excluded from auto-deployment.

#Configuration and Cluster Keys

#The Ceph Config File

The main config lives in /etc/ceph/ceph.conf. This file contains global and per-daemon sections, such as the cluster name, mon IPs, and public/cluster networks. For dynamic runtime configuration, it's better to use the config database (ceph config set) rather than editing the file.

ceph config get mon
ceph config dump

ceph config dump shows all currently applied configuration values and their sources — from defaults, files, or explicit sets. This way you know why a parameter has a given value.

#Cluster Keys and CephX

Every daemon and client uses a keyring for authentication via CephX. Keyrings are stored in /etc/ceph, and the bootstrap key is deployed automatically during bootstrap or when adding hosts.

ceph auth ls

ceph auth ls shows all entities and their capabilities. Auth and capability management will be covered in depth in episode 10.

#Verifying Cluster Health

#Understanding the ceph status Output

Once the OSDs are running, do a full verification:

ceph status
ceph health detail
ceph df

ceph status shows the mon quorum, the active mgr, the number of OSDs, and the PG status. ceph df shows capacity and usage per pool. The cluster is considered healthy when the status is HEALTH_OK and no PG is incomplete or degraded.

ceph -w

ceph -w monitors cluster events in real time. When the cluster is newly created, you'll see PGs move from creating to active+clean. If all PGs are active+clean, the cluster is ready for use.

#Conclusion

In this episode you've built a real three-node Ceph cluster: preparing the user and SSH, bootstrapping with cephadm, adding MONs, MGRs, and OSDs, understanding the config file and CephX keyrings, and verifying cluster health until HEALTH_OK.

The key takeaways:

• Set up passwordless SSH and passwordless sudo before bootstrapping.
• cephadm bootstrap creates the first MON, MGR, and keyring.
• Add up to three MONs for a safe quorum.
• ceph orch apply osd --all-available-devices deploys all empty disks.
• Runtime config is managed via ceph config set, keys via CephX.
• ceph status and ceph -w are the primary verification tools.

In the next episode, episode 4, we'll move into RADOS and pool management — creating pools, setting replication levels, understanding the replicated vs erasure coded pool types, tuning placement groups and failure domains, and managing the OSD lifecycle. Make sure your cluster is healthy before continuing!