Learn Ceph - File Storage with CephFS

#Introduction

Welcome to episode 6 of the Learn Ceph series! After RBD for block storage, it's time for the second interface: CephFS, file storage with POSIX semantics. CephFS is Ceph's oldest interface and the primary choice for workloads that need a shared filesystem.

Unlike RBD, which is accessed as a raw device, CephFS appears as directories and files that can be mounted on many clients at once with strong consistency. That's what makes it suitable for home directories, shared workspaces, and applications that need file access.

In this episode we'll cover the CephFS architecture involving the metadata server, creating a filesystem and mounting clients, active-active MDS for scaling metadata, and quotas, snapshots, and client consistency.

#CephFS Architecture: Metadata Server and Data Path

#The Role of the MDS

The MDS (Metadata Server) stores CephFS metadata: file names, directories, permissions, and layouts. Metadata is stored in a dedicated cephfs_metadata pool, while file data is stored in the cephfs_data pool. Both are pools within RADOS.

The MDS doesn't store file data at all. When a client reads a file, it asks the MDS for metadata, then reads the data directly from the OSDs. This design lets CephFS scale without a bottleneck on a single file server.

#The File Access Flow

The flow when a client reads a file:

1. The client asks the MDS for the file's metadata location and capabilities.
2. The MDS returns the inode, permissions, and layout of the file.
3. The client reads the file data directly from the OSDs via RADOS.
4. To write, the client sends data to the OSDs and notifies the MDS of metadata changes.

client --> MDS (metadata + caps)
client --> OSD (data directly)

#Creating a Filesystem and Mounting Clients

#Creating Pools and the Filesystem

The first step is creating two pools for metadata and data, then creating the filesystem:

ceph osd pool create cephfs_data 128
ceph osd pool create cephfs_metadata 64
ceph fs new myfs cephfs_metadata cephfs_data
ceph fs ls

ceph fs new myfs cephfs_metadata cephfs_data creates a filesystem named myfs with metadata and data pools. Note that the metadata pool should use fast OSDs because metadata latency is highly sensitive.

#Mounting CephFS on a Linux Client

To mount on a Linux client, use the kernel driver or FUSE. The kernel driver is faster, while FUSE is more flexible:

sudo mkdir /mnt/myfs
sudo mount -t ceph mon-node1:6789:/ /mnt/myfs \
  -o name=admin,secretfile=/etc/ceph/admin.secret

sudo mount -t ceph needs the client's CephX secret. A safer approach is using the /etc/ceph/ceph.client.<user>.keyring file and defining the mount in /etc/fstab.

#The FUSE Alternative

If the kernel driver isn't available, CephFS can be mounted through ceph-fuse:

ceph-fuse /mnt/myfs -o rw

ceph-fuse runs as a user-space process and doesn't require a kernel module. FUSE performance is slightly lower than the kernel driver, but deployment is much easier across different distros.

#Active-Active MDS and Metadata Scaling

#More Than One Active MDS

CephFS supports active-active MDS: several active MDSs work in parallel, each handling a different portion of the metadata. Standby and standby-replay MDSs stay on alert and take over immediately when an active MDS fails.

ceph fs set myfs max_mds 3
ceph orch apply mds myfs 3

ceph fs set myfs max_mds 3 sets up to three active MDSs. With three MDSs, metadata is distributed across more processes, so metadata operation throughput increases.

#Monitoring MDS Load

Monitoring metadata distribution is important to make sure scaling works:

ceph fs status myfs
ceph mds stat

ceph fs status shows the list of MDSs, ranks, states, and the pools in use. If one MDS becomes a hot spot, Ceph automatically rebalances the metadata sub-trees through its balancing mechanism.

#Quotas, Snapshots, and Client Consistency

#Applying Quotas

CephFS supports quotas based on capacity and file count on directories:

sudo setfattr -n ceph.quota.max_bytes -v 10G /mnt/myfs/project
sudo setfattr -n ceph.quota.max_files -v 10000 /mnt/myfs/project

ceph.quota.max_bytes and ceph.quota.max_files limit the usage of the project directory. Quotas are enforced when a client requests a block allocation or creates an inode, so no separate daemon is required.

#Creating CephFS Snapshots

CephFS snapshots are created per directory through the .snap mechanism:

sudo mkdir /mnt/myfs/project/.snap/snap-before-upgrade
ls /mnt/myfs/project/.snap/

Creating a .snap/<name> directory inside the target directory directly produces a read-only snapshot. This feature must be enabled with ceph fs set myfs allow_new_snaps true if it isn't already.

#Client Consistency

CephFS uses capability-based caching: clients cache metadata and data while holding capabilities from the MDS. When another client wants to write the same file, the MDS revokes the capability so the client performs a write-through.

ceph fs dump myfs
ceph tell mds.<rank> session ls

ceph tell mds.<rank> session ls shows the active client sessions. This consistency provides strong POSIX semantics, but keep in mind that CephFS isn't designed for files accessed by many clients concurrently with high write rates on a single file.

#Conclusion

In this episode you've understood and practiced CephFS: the MDS architecture and data path, creating a filesystem and mounting with the kernel driver or FUSE, active-active MDS for metadata scaling, and quotas, snapshots, and the client consistency mechanism.

The key takeaways:

• CephFS separates metadata (MDS) and data (OSD) into different pools.
• Mount with mount -t ceph for the kernel driver or ceph-fuse for user-space.
• Active-active MDS distributes metadata so throughput rises with the number of MDSs.
• Quotas use ceph.quota.max_bytes and ceph.quota.max_files.
• Snapshots are created with .snap directories and are read-only.
• Consistency comes from capability-based caching managed by the MDS.

In the next episode, episode 7, we'll cover object storage with RGW — the RADOS Gateway concepts and the S3/Swift API, setting up realms, zones, and zonegroups, bucket, object, and user management, and integration with S3 clients and object lifecycle. The last interface in your unified storage trio!