Learn Ceph - Core Concepts & Architecture

#Introduction

Welcome to episode 2 of the Learn Ceph series! In episode 1 you understood why Ceph is the right choice. Now we move into the part that's most important to understand fully: Ceph's internal architecture. This episode dissects the components running inside the cluster and how they work together.

Understanding Ceph's architecture isn't just a formality. When problems arise later — an OSD going down, stuck PGs, or slow rebalancing — you'll need to know which part is responsible. Without this mental map, Ceph operations will feel confusing.

In this episode we'll cover the core components MON, OSD, MDS, MGR, and RGW, the RADOS foundation, how the CRUSH map works, and the concepts of pools, placement groups, replication, and erasure coding. Let's get started.

#Ceph's Core Components

#MON (Monitor) — The Cluster's Source of Truth

MON maintains the cluster maps: monitor map, OSD map, MGR map, CRUSH map, and metadata. Every change must go through consensus via the MON quorum. That's why a minimum of three MONs is recommended — if two of the three MONs die, quorum is lost and the cluster stops accepting changes.

MON doesn't store user data. Its job is only to maintain cluster state so that all daemons and clients see the same picture.

#OSD (Object Storage Daemon) — Where Data Lives

OSD stores data, handles replication, recovery, and rebalancing, and answers read/write requests from clients. One OSD is usually tied to one disk. The OSD is also responsible for checking the heartbeat of other daemons and reporting their status to the MON.

#MGR (Manager) — The Operational Brain

MGR collects metrics and state from the entire cluster, provides the web dashboard, and exposes a Prometheus endpoint. Modern MGRs also handle balancing, deep scrub scheduling, and integration with tooling like cephadm. MGR runs in active-standby mode.

#MDS (Metadata Server) — The Brain of CephFS

MDS stores metadata for CephFS: file names, directories, permissions, and the hierarchy structure. MDS doesn't store file data; the data stays in RADOS. Multiple MDSs can run in active-active mode to scale metadata throughput.

#RGW (RADOS Gateway) — The Door to Object Storage

RGW provides object storage compatible with S3 and Swift. RGW translates HTTP requests into RADOS operations, including bucket, user, and access policy management.

#RADOS: The Foundation of All Services

#What Is RADOS

RADOS (Reliable Autonomic Distributed Object Store) is the layer where all Ceph data is stored. Every object in Ceph — an RBD image, a CephFS file, an S3 object — is ultimately stored as a RADOS object in a pool.

The advantage of RADOS is that all placement, replication, recovery, and rebalancing logic lives at this level. The interfaces above it (RBD, CephFS, RGW) only manage how data is read and written.

RBD (block)  CephFS (file)  RGW (object)
        \        |         /
         v       v        v
              RADOS
         MON  MGR  MDS  OSD

#Reliability Comes from the Bottom Up

Ceph's reliability doesn't come from a single piece of hardware, but from replication at the object level. When a client writes an object, the OSD stores several copies on different nodes according to the CRUSH rule. If one copy fails, the other copies keep serving requests while the cluster repairs itself.

#CRUSH Map and Data Placement

#The CRUSH Algorithm

CRUSH (Controlled Replication Under Scalable Hashing) is the algorithm that determines where objects are stored, based on the CRUSH map and a hash of the object. Clients compute data locations on their own — there's no central lookup service, so there's no bottleneck.

The CRUSH map describes the cluster hierarchy: root, failure domains (host, rack), and the OSDs within them. By configuring rules, you control that replicas are never placed on the same host or rack.

#Failure Domain and Rules

ceph osd tree
ceph osd crush rule ls
ceph osd crush rule dump replicated_rule

ceph osd tree shows the hierarchy of nodes and OSDs along with their status, while ceph osd crush rule dump shows the default placement rule. The default failure domain is usually the host, meaning the replicas of an object are placed on different hosts.

#Pool, Placement Group, and Replication

#Pools and Placement Groups

A pool is a logical partition where objects are stored. Each pool has a number of placement groups (PGs) determined at creation time. A PG is a logical shard of the pool: every object is mapped to one PG, and every PG is mapped to several OSDs.

ceph pg stat
ceph osd pool ls detail

The number of PGs determines the granularity of rebalancing. Too few PGs cause uneven distribution; too many overload the MON and OSDs. For a pool holding 100 GB across 100 OSDs, a good rule of thumb is about 100 PGs per OSD across the cluster.

#Replication vs Erasure Coding

There are two ways to maintain data reliability in a pool:

• Replicated: each object is copied whole into size copies (usually 3). Simple, with 3x disk usage.
• Erasure coded: data is split into k+m chunks, and only k are needed to read. Saves disk at the cost of higher CPU overhead.

ceph osd pool create data 128
ceph osd pool set data size 3

ceph osd pool create data 128 creates a replicated pool with 128 PGs and the default replica size. We'll dig deep into the size and min_size settings in episode 4.

#Conclusion

In this episode you've understood the Ceph architecture map: the MON, OSD, MDS, MGR, and RGW components and their roles, RADOS as the single foundation, the CRUSH map for data placement without a central index, and the concepts of pools, PGs, replication, and erasure coding.

The key takeaways:

• MON maintains cluster state and needs a majority quorum to function.
• OSD stores and replicates data; MGR manages metrics and the dashboard.
• MDS serves CephFS metadata, RGW provides the S3/Swift API.
• All Ceph data is stored as RADOS objects in a pool.
• CRUSH places replicas with failure domains so they don't die together.
• Replicated relies on whole copies; erasure coding saves disk at the cost of CPU overhead.

In the next episode, episode 3, we'll get hands-on: installation and basic cluster setup — preparing the cluster with cephadm, configuring MON, OSD, and MGR, understanding the config file and cluster keys, and verifying health with ceph status. Get your three nodes ready, because we're starting to build!