Learn Ceph - Ceph Client Integration

#Introduction

Welcome to episode 8 of the Learn Ceph series! You've gotten to know the three main interfaces — RBD, CephFS, and RGW — in episodes 5 through 7. Now it's time to answer a more practical question: how do you connect clients to the cluster from the user side? This episode covers Ceph client integration comprehensively.

A Ceph cluster is useless without clients. A client can be a Linux server that needs RBD disks, a workstation mounting CephFS, or an application accessing S3 buckets through RGW. Each client type has different paths, credentials, and configurations, even though everything runs on top of RADOS.

By the end of this episode you'll be able to configure Linux clients for CephFS and RBD, set up RGW clients securely, understand how the Kubernetes CSI drivers work, and apply client authentication and keyring best practices. Let's get started.

#Connecting Linux Clients to CephFS and RBD

#Setting Up the Keyring and Ceph Config on the Client

Every client needs two things: the /etc/ceph/ceph.conf config file and the matching keyring. The easiest way is copying both from a cluster node:

scp node1:/etc/ceph/ceph.conf /etc/ceph/
scp node1:/etc/ceph/ceph.client.admin.keyring /etc/ceph/

The ceph.conf file contains the mon list and network parameters, while the keyring holds the credentials. Using the admin keyring on general clients is practical, but it's not a safe practice for production — we'll create dedicated client keys in the authentication section.

#Mounting CephFS from a Client

Once the config and keyring are available, mount CephFS with the kernel driver:

sudo mkdir -p /mnt/myfs
sudo mount -t ceph ceph-mon1:6789:/ /mnt/myfs \
  -o name=admin,secretfile=/etc/ceph/admin.secret

The mount -t ceph syntax needs the MON list and credentials. For persistence across reboots, define an entry in /etc/fstab with the same options. If the kernel driver isn't available, use ceph-fuse as a user-space alternative.

#Mapping an RBD Image from a Client

For RBD, the client needs the ceph-common and rbd packages, then maps the image:

dnf install -y ceph-common
rbd map rbd-pool/vol-data --id admin

The rbd map command maps the image into a /dev/rbdN device. This device can be partitioned, formatted, and mounted like a regular disk. Don't forget to run rbd unmap /dev/rbdN before shutting down the node so no writes are left behind.

#Configuring RADOS Gateway Clients

#S3 Clients with aws cli

RGW clients don't need ceph.conf because they communicate over HTTP. All they need is the endpoint, access key, and secret key from the RGW user created in episode 7:

aws configure --profile rgw
export AWS_PROFILE=rgw
aws --endpoint-url https://rgw.example.com s3 ls

The aws configure command stores credentials in ~/.aws/credentials. All s3 operations are then directed at RGW with --endpoint-url. For production, consider short-lived credentials or instance roles if your cloud platform supports them.

#Client Libraries in Applications

Applications use SDKs like boto3 in Python or aws-sdk-go in Go. The key point: set the endpoint to RGW instead of the default AWS endpoint. That way, applications already written for S3 work without any business logic changes.

boto3 client for RGW

import boto3
 
s3 = boto3.client(
    "s3",
    endpoint_url="https://rgw.example.com",
    aws_access_key_id="AKIAEXAMPLE",
    aws_secret_access_key="SECRETEXAMPLE",
)
 
for obj in s3.list_objects_v2(Bucket="first-bucket").get("Contents", []):
    print(obj["Key"])

The endpoint_url="https://rgw.example.com" line is the only difference from a regular S3 client. Make sure the endpoint uses TLS when crossing public networks.

#Integration with Kubernetes CSI Drivers

#The CSI Concept in Kubernetes

The Container Storage Interface (CSI) lets Kubernetes provision external storage automatically. For Ceph, there are two drivers: RBD CSI for block volumes and CephFS CSI for shared filesystems. Both talk to the Ceph cluster using keyrings managed as Kubernetes Secrets.

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
stringData:
  userID: kubernetes
  userKey: <base64-keyring>
  adminID: admin
  adminKey: <base64-admin-key>

The Secret above holds the credentials the drivers use. The full StorageClass and PVC deployment will be covered in episode 16, but the basic concept is already visible: the driver uses the credentials from the Secret to talk to the MONs.

#Integration with Cloud Platforms

On cloud platforms, Ceph often becomes the native storage backend. The most famous example is OpenStack: Cinder uses RBD for block volumes, and Manila can use CephFS for the shared file service. Configuration happens on the cloud service side, while the Ceph cluster only needs to provide the appropriate pool and keyring.

ceph auth get-or-create client.cinder mon 'allow r' \
  osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes'

The command above creates a dedicated client for Cinder with capabilities restricted to only the volumes pool. This least-privilege principle is the main best practice we'll cover next.

#Client Authentication and Keyring Best Practices

#One Keyring per Client

Avoid using the admin keyring on production clients. Create one CephX entity per client or per workload group, with capabilities covering only what's needed:

ceph auth get-or-create client.backup mon 'allow r' \
  osd 'allow rwx pool=backups'
ceph auth export client.backup -o /etc/ceph/ceph.client.backup.keyring

ceph auth get-or-create creates a new key and exports it to a keyring file at the same time. Rotate keys periodically with ceph auth rotate-key, especially when a team member leaves.

#Protecting Keyring Files

Keyring files contain secrets and must be protected like any other credential file:

chown root:ceph /etc/ceph/ceph.client.*.keyring
chmod 640 /etc/ceph/ceph.client.*.keyring

The chmod 640 permissions ensure only the owner and group can read the keyring. Never put keyrings in publicly readable directories, and avoid storing them unencrypted in container images.

#Conclusion

In this episode you've understood how to connect various clients to the Ceph cluster: mounting CephFS and mapping RBD from Linux clients, configuring RGW clients with aws cli and SDKs, integrating with Kubernetes CSI drivers and cloud platforms like OpenStack, and client authentication and keyring management best practices.

The key takeaways:

• Linux clients need ceph.conf and a keyring before they can access CephFS or RBD.
• RGW clients only need an endpoint, access key, and secret key — no ceph.conf.
• Kubernetes CSI uses Secrets to store the RBD and CephFS driver credentials.
• Cinder and Manila use dedicated keyrings restricted per pool.
• One CephX entity per workload with minimal capabilities (least privilege).
• Keep keyring permissions private and rotate keys periodically.

In the next episode, episode 9, we'll move into performance tuning & data placement — optimizing OSDs with BlueStore and DB/WAL, tuning the network with public and private network separation, adjusting placement groups and device classes, and monitoring throughput, latency, and backfill behavior. Get your performance measurement tools ready!