Learn Ceph - Extending Ceph Ecosystem

#Introduction

Welcome to episode 18 of the Learn Ceph series! So far you've used Ceph's built-in features. But a platform's real power lies in its ecosystem: additional services, automation tooling, and the community that keeps developing it. This episode covers extending the Ceph ecosystem.

Advanced features like RBD mirroring, bucket notifications, and Ansible and Terraform integration turn Ceph from mere storage into a platform that can be operated as code. You'll also see how custom monitoring tooling is built on top of the metrics that are already exposed.

By the end of this episode you'll understand how to use Ceph's additional services, automate operations with Ansible and Terraform, build custom tooling, and give back to the Ceph open-source project. Let's get started.

#Additional Services: Mirroring, Snapshots, and Notifications

#RBD Mirroring for Cross-Cluster Recovery

RBD mirroring replicates RBD images to another Ceph cluster asynchronously (or synchronously). This provides disaster recovery at the block storage level, distinct from RGW multisite which handles object storage:

rbd mirror pool enable rbd-pool image
rbd mirror pool info rbd-pool

rbd mirror pool enable enables image mirroring mode on the pool. Once the peer cluster is connected, every marked image is replicated to the peer.

#Snapshots and Integrated Scheduling

Combining scheduled snapshots with mirroring provides more granular recovery:

rbd mirror snapshot schedule add --pool rbd-pool \
  --image vol-data 1h
rbd mirror snapshot schedule list --pool rbd-pool

rbd mirror snapshot schedule add creates a snapshot every hour for the vol-data image. These scheduled snapshots are also replicated and become regular recovery points.

#RGW Bucket Notifications

Bucket notifications connect object storage to external systems. When an object is created, deleted, or modified, RGW sends an event to a message broker:

radosgw-admin topic create --name=data-baru \
  --endpoint=kafka://kafka.example.com:9092

radosgw-admin topic create defines a topic pointing at a Kafka broker. The topic is then attached to a bucket so every object event triggers a notification to Kafka.

#Integration with Ansible, Terraform, and Automation

#Automating Ceph with Ansible

Although cephadm handles daemon orchestration, initial node configuration and integration with other systems are often automated with Ansible. A simple playbook to ensure a node's basic configuration:

- name: Siapkan node Ceph
  hosts: ceph_nodes
  become: true
  tasks:
    - name: Tambahkan user ceph
      user:
        name: ceph
        groups: wheel
    - name: Salin key SSH
      authorized_key:
        user: ceph
        key: "{{ lookup('file', 'id_rsa.pub') }}"

authorized_key ensures the bootstrap SSH key is spread to all nodes. With Ansible, provisioning 50 nodes isn't much different from 3 nodes.

#Provisioning with Terraform

Terraform handles infrastructure provisioning — VMs, disks, network — before Ceph is installed on top:

resource "openstack_compute_instance_v2" "storage_node" {
  name      = "ceph-node-1"
  flavor    = "storage-flavor"
  image     = "rocky9"
  count     = 3
}

openstack_compute_instance_v2 creates three storage VMs with a dedicated flavor. Combining Terraform for infrastructure and Ansible for configuration gives you a repeatable provisioning pipeline.

#Custom Monitoring and Operational Tooling

#Building Custom Metrics

Besides built-in metrics, Ceph lets you expose custom metrics from MGR modules or scripts that call the API. This enables dashboards that reflect business context, not just cluster health:

curl -s http://mgr-host:9283/metrics \
  | grep ceph_pool_quota_bytes_used

ceph_pool_quota_bytes_used is an example of a per-pool metric that can be the basis for quota usage alerts. Build queries in Grafana for any metric available at the Prometheus endpoint.

#Your Own Operational Tooling

Many teams build internal scripts for repetitive tasks: daily capacity reports, user creation automation, or ticketing system integration. The ceph CLI and radosgw-admin use formats that are easy to parse, so integrating with internal scripts and APIs is relatively easy.

#Contributing to the Open-Source Ecosystem

#How to Contribute

Ceph is an open-source project with an active community. Contributions don't have to be code — documentation, test reports, or helping answer questions all count. For code contributions, start small: fix a bug, add a test, or improve documentation.

git clone https://github.com/ceph/ceph.git
cd ceph

git clone copies the Ceph source. Before contributing, read the contributor guide at docs.ceph.com and start with an issue labeled good first issue — it's a friendly entry point for newcomers.

#Following the Community

The Ceph community is active on the ceph-users and ceph-devel mailing lists, the official forum, and IRC channels. Following these discussions not only broadens your insight, but also expands your network and opens career opportunities in storage. The official documentation and release changelogs are also inexhaustible learning resources.

#Conclusion

In this episode you've understood how to extend the Ceph ecosystem: using additional services like RBD mirroring, scheduled snapshots, and bucket notifications, automating operations with Ansible and Terraform, building custom monitoring tooling on top of Prometheus metrics, and giving back to the Ceph open-source project.

The key takeaways:

• RBD mirroring replicates images to another cluster for block storage recovery.
• Scheduled snapshots and mirroring work together to provide regular recovery points.
• Bucket notifications connect RGW to Kafka or other brokers.
• Ansible automates node configuration, Terraform automates infrastructure provisioning.
• Prometheus metrics can be turned into custom dashboards and tooling.
• Contributing to Ceph can start with documentation and small bugs.

In the next episode, episode 19, we'll cover operational readiness & runbooks — writing runbooks for incidents and cluster health recovery, managing maintenance windows, upgrades, and rolling updates, preparing disaster recovery procedures and training, and building operational documentation and team knowledge. Time to turn skills into repeatable processes!