Learn Ceph - Operational Readiness & Runbooks

#Introduction

Welcome to episode 19 of the Learn Ceph series! Technical skill alone isn't enough to operate storage used by many teams. What separates professional operations from mere tinkering is readiness: clear runbooks, tested processes, and a trained team. This episode covers operational readiness & runbooks.

When the cluster has problems at 2 a.m., the best reaction isn't racking your brain over tutorials — it's opening a prepared runbook and following its steps. A good runbook turns a stressful incident into a checklist you can execute calmly.

By the end of this episode you'll be able to write runbooks for common scenarios, run maintenance and upgrades safely, prepare recovery procedures and team training, and build living operational documentation. Let's get started.

#Writing Runbooks for Incidents

#Anatomy of a Good Runbook

A good runbook has a fixed structure for every scenario: symptoms, common causes, diagnosis steps, mitigation steps, and completion criteria. Start with the most frequent scenarios: HEALTH_WARN cluster, OSD down, stuck PGs, high latency, and low capacity.

ceph status
ceph health detail
ceph osd tree

These three commands are the standard opening for any diagnosis. ceph status gives a summary, ceph health detail explains the reason for a warning, and ceph osd tree checks OSD status.

#Example Runbook: OSD Down

Take the OSD down runbook as an example. Its steps:

1. Identify which OSD is down from ceph osd tree.
2. Check the OSD log for the cause: disk error, network, or crash.
3. If temporary, wait for the OSD to come back or restart the daemon.
4. If the disk is damaged, follow the replacement procedure from episode 12.
5. Verify PGs return to active+clean.

A runbook like this is written once, reused repeatedly, and updated whenever an incident teaches something new.

#Maintenance Windows and Upgrades

#Managing Maintenance Windows

Maintenance requires planning: pick quiet hours, notify workload owners, and make sure no other changes run at the same time. For operations affecting many OSDs, use commands that reduce risk:

ceph osd set noout

ceph osd set noout prevents Ceph from marking OSDs as out during maintenance restarts, so unnecessary data rebalancing doesn't happen. Don't forget ceph osd unset noout when you're done.

#Rolling Upgrades

Ceph upgrades are done gradually, daemon by daemon, so service never stops completely. With cephadm, an upgrade is just triggered to a target version:

ceph orch upgrade start --ceph-version 19.2.0
ceph orch upgrade status

ceph orch upgrade start performs an automatic rolling upgrade: MONs first, then MGRs, then OSDs one by one. ceph orch upgrade status monitors the progress. During the upgrade, watch cluster health and prepare a rollback if needed.

#Disaster Recovery Procedures and Training

#Tested Recovery Procedures

Disaster recovery procedures only matter if they've been tested. Keep procedures for every major scenario: losing one node, losing one site (if multisite), data corruption, and accidental deletion. Each procedure references a runbook and clear verification steps.

ceph status
ceph health detail
ceph fs status myfs

ceph fs status verifies that the CephFS filesystem recovered after recovery. Make this verification sequence part of the end of every recovery procedure.

#Training the Team

A runbook that's never been practiced will fail in real use. Schedule regular training sessions: game day simulations where a team member kills an OSD in the staging environment and the team follows the runbook to recover it. These exercises expose holes in procedures before they become real incidents.

#Documentation and Team Knowledge Base

#Living Documentation

Operational documentation should live alongside the infrastructure: cluster architecture, pool and rule list, credential map, workload owner contacts, and change log. Keep it somewhere that can be reviewed and updated — for example, a shared Git repository along with the configuration.

git init
git add docs/ ansible/ ceph.conf
git commit -m "docs: dokumentasi arsitektur cluster"

git init turns documentation into a versioned artifact. Every operational change is recorded and traceable, including by new team members.

#A Knowledge Base from Incidents

After every incident, write a short postmortem: what happened, how it was detected, how it was handled, and what would be done better. This collection of postmortems becomes your most valuable knowledge base — far more relevant than generic tutorials because it's born from your own infrastructure.

#Conclusion

In this episode you've understood how to build operational readiness for a Ceph cluster: writing structured runbooks for common incidents, managing maintenance windows and rolling upgrades safely, preparing tested disaster recovery procedures and team training, and building living operational documentation and a knowledge base from incidents.

The key takeaways:

• Structured runbooks turn incidents into executable checklists.
• ceph osd set noout protects data during maintenance.
• cephadm handles automatic rolling upgrades with monitorable status.
• Recovery procedures must be tested through regular drills and game days.
• Architecture and configuration documentation is stored in Git alongside code.
• Postmortems from each incident become the team's knowledge base.

In the next episode, episode 20, we'll cover real-world use cases & patterns — case studies of cloud block storage, enterprise file services, and S3-compatible object storage, using Ceph for OpenStack, Kubernetes, backups, and media, architectural patterns for scale and multi-tenancy, and cost and capacity planning. Time to see Ceph working in the real world!