Learn Ceph - Monitoring & Observability

#Introduction

Welcome to episode 11 of the Learn Ceph series! Your cluster is secure and fast, but one big question remains: how do you know what's happening at any moment? This episode covers monitoring & observability — making your Ceph cluster transparent so problems become visible before they turn into disasters.

Observability isn't just a nice dashboard. It's the ability to answer questions like: why did latency go up? Which OSD is starting to slow down? How much capacity is left for pool X? Without good metrics and logs, all of those questions can only be answered by guessing.

By the end of this episode you'll be able to use the Ceph dashboard, collect metrics via Prometheus, assemble Grafana dashboards, configure alerting, and read logs for troubleshooting and audit purposes. Let's get started.

#Ceph Dashboard

#Enabling and Accessing the Dashboard

The Ceph dashboard is integrated directly with cephadm and is active by default after bootstrap. For deployments that haven't enabled it yet:

ceph mgr module enable dashboard
ceph mgr services
ceph dashboard create-self-signed-cert

ceph mgr module enable dashboard enables the dashboard module in the MGR, while ceph mgr services shows the accessible URL. The dashboard usually runs on port 8443 over HTTPS.

#Main Dashboard Features

The dashboard presents a complete cluster overview in one page: health, capacity per pool, OSD status, client activity, and performance metrics such as IOPS and latency. You can also perform a few basic operations like viewing daemon logs and checking PG status.

ceph dashboard ac-user-set-password admin password-baru

ceph dashboard ac-user-set-password changes the dashboard user password. Change the default password immediately after bootstrap and restrict dashboard access to the internal network only.

#Prometheus Exporter and Metrics

#Using the Prometheus Module

Ceph provides a Prometheus endpoint through the MGR. Enable the module, then verify the metrics endpoint:

ceph mgr module enable prometheus
curl -s http://mgr-host:9283/metrics | head

ceph mgr module enable prometheus runs the exporter on port 9283 on every active MGR. The /metrics endpoint serves metrics in a format Prometheus understands, including ceph_cluster_total_used_bytes and ceph_osd_up.

#Key Metrics to Monitor

Metrics you should always monitor include health status, the number of PGs in a degraded or stale state, OSD commit latency, IOPS and throughput, capacity per pool, and the object count. These metrics become the foundation of the alerts we'll assemble.

#Grafana Dashboards

#Deploying Grafana with cephadm

cephadm can deploy Grafana automatically together with the monitoring stack:

ceph orch apply prometheus
ceph orch apply grafana
ceph orch apply alertmanager

ceph orch apply prometheus deploys Prometheus, Grafana, and Alertmanager as managed daemons. Ceph automatically provides built-in Grafana dashboards already configured to display Ceph metrics.

#Reading the Key Dashboards

The built-in Ceph Grafana dashboards have panels for capacity, IOPS, latency, and PG health. When an anomaly occurs, the latency and PG panels show a distinctive pattern: for example, latency rising together with the number of degraded PGs — that's an indication recovery is in progress.

Ceph daemons --> Prometheus exporter --> Prometheus --> Grafana
                                          |--> Alertmanager --> notifikasi

The flow above shows the journey of metrics from Ceph daemons to alert notifications. Grafana is for visualization, Alertmanager is for sending notifications.

#Alerting and Performance Anomalies

#Building Alert Rules

Good alerts only warn about things you can act on. Common basic rules:

• CephHealthError: cluster in HEALTH_ERR for more than one minute.
• CephPGsInactive: there are inactive or stale PGs.
• CephOSDDownHigh: more than one OSD down for a long time.
• CephMonDownQuorumAtRisk: MON down approaching the quorum limit.

Alert rules are written in Prometheus format. A simple example for an OSD down:

groups:
  - name: ceph
    rules:
      - alert: CephOSDDown
        expr: ceph_osd_up == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "OSD {{ $labels.osd }} down selama 10 menit"

The rule above triggers an alert if an OSD has been down for more than 10 minutes. Place the rule file in the directory Prometheus scrapes and reload the configuration.

#Managing Notifications

Alertmanager routes notifications to email, Slack, or webhooks according to the alertmanager.yml configuration. Ceph uses configuration placed in ceph orch alertmanager config to inject routes and receivers. Start with one receiver to your team's chat channel, then add escalation for higher severities.

#Log Management and Audit Trails

#Ceph Daemon Logs

Every daemon produces logs stored in /var/log/ceph on its respective node. OSD logs are the most common troubleshooting source because they contain I/O errors and peering events:

journalctl -u ceph-osd@0 --since "10 min ago"

journalctl -u ceph-osd@0 reads OSD 0's log from the systemd journal. To search for errors, filter with keywords like ERR or WARN. Centralized audit logging can be built by forwarding the journal to a forwarder like Loki or an external syslog.

#Audit Trails

For compliance purposes, enable audit logging on RGW that records operations per user and bucket:

ceph config set rgw rgw_log_operations true
ceph config set rgw rgw_log_http_headers true

rgw_log_operations records every object and bucket operation to the log. With this audit trail, you can answer the question "who accessed which object and when" — something that's mandatory for multi-tenant storage in audited environments.

#Conclusion

In this episode you've understood how to build observability for a Ceph cluster: using the Ceph dashboard, collecting metrics with the Prometheus module, assembling Grafana dashboards, configuring alerting for cluster events and performance anomalies, and managing daemon logs and audit trails for troubleshooting and compliance.

The key takeaways:

• The Ceph dashboard gives a complete visual overview of health and performance.
• The Prometheus module exposes metrics on port 9283 in Prometheus format.
• cephadm can deploy Prometheus, Grafana, and Alertmanager automatically.
• Good alert rules focus on actionable things, like OSD down.
• Daemon logs can be read via journalctl and filtered with keywords.
• RGW audit logs record operations per user for compliance purposes.

In the next episode, episode 12, we'll cover high availability & fault tolerance — redundancy for MON, MGR, MDS, and RGW, CRUSH failure domains and host or device isolation, handling OSD failure and disk replacement, and heal, backfill, and scrub operations. Time to make your cluster resilient!