Learn OpenStack - Monitoring, Logging & Alerting (Prometheus, Grafana & ELK)

#Introduction

A healthy production cluster can't be left running without eyes. In episode 17 you secured it; episode 18 makes it observable — so failures are detected before they become downtime, and root causes are found in minutes instead of days. That's the job of observability, and in OpenStack it's built from three pillars: monitoring, logging, and alerting.

Episode 18 covers all three concretely: Prometheus and exporters for service metrics, Grafana for dashboards, the ELK Stack or Grafana Loki for centralized logs, and Alertmanager for proactive notifications when critical conditions occur.

#Monitoring OpenStack Services

#Prometheus Exporters

Prometheus collects metrics from exporters — agents that expose metrics in a format Prometheus understands. Every OpenStack layer has its exporter:

OpenStack service → exporter → scrape (Prometheus) → Grafana
                                                    → Alertmanager

openstack-exporter reads service APIs and exposes metrics such as instance count, service status, and response time — the main data source for dashboards.

#Key Metrics to Monitor

Grafana becomes the visual window into all metrics. A production OpenStack dashboard usually contains:

• API Response Times: endpoint latency per service.
• Instance Count: active and in-progress instance counts.
• Hypervisor CPU/RAM Usage: utilization of each compute node.
• Cinder Volume Capacity: space usage of the volume backend.
• RabbitMQ Queue Length: message buildup, a signal of trouble.

openstack_exporter_instances{state="ACTIVE"}

The command above is an example PromQL query — it surfaces the active instance count per project. With a Grafana dashboard, these metrics are visualized on a single screen.

#Centralized Logging

#The ELK Stack

All OpenStack services write logs to their own files — searching for errors across dozens of nodes means opening dozens of files. The solution is centralized logging: collect all logs into one place.

The ELK Stack is the classic combination:

• Elasticsearch: log storage and search.
• Logstash/Fluentd: the pipeline that collects and processes logs.
• Kibana: the search and visualization interface.

curl -s 'http://kibana.example.com/api/console/proxy?path=_search&method=GET' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"match":{"message":"ERROR"}}}'

A modern alternative is Grafana Loki — a log store that integrates directly with the Grafana ecosystem, using labels instead of indexes, so it's lighter to operate.

#Important Logs to Centralize

Prioritize logs from: nova-compute (instance spawn errors), neutron-server and agents (network issues), cinder-volume (attach failures), and keystone (failed login attempts). With centralized logs, searching for errors across services becomes a single command.

#Alerting

#Alertmanager and Rules

Metrics and logs are useless if no one follows up. Alertmanager turns metrics into notifications — email, Slack, or webhook. Alert rules are defined as Prometheus rules:

groups:
  - name: openstack.rules
    rules:
      - alert: ComputeNodeDown
        expr: up{job="node"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Compute node tidak merespons"

The rule above fires ComputeNodeDown when a node doesn't answer a scrape for 5 minutes. for: 5m prevents false alerts from momentary spikes.

#Important Alert Scenarios

The three alerts that most often save production:

• Compute Node Down: instances can't be scheduled or migrated.
• RabbitMQ Queue Overflow: messages pile up — often a sign of a slow service or a dropped connection.
• Ceph OSD Degraded: replicas are reduced — data is at risk if it doesn't recover soon.

sudo docker exec rabbitmq rabbitmqctl list_queues name messages_ready

The sudo docker exec rabbitmq rabbitmqctl list_queues command inspects the message queues directly — if messages_ready keeps climbing, check the dependent service before the alert starts ringing.

#Summary

Episode 18 builds the cluster's eyes and ears: Prometheus and exporters for service metrics, Grafana for dashboards of API response, instance count, and hypervisor utilization, the ELK Stack or Loki for centralized logs, and Alertmanager with rules that fire notifications when a compute node is down, a queue overflows, or an OSD is degraded.

Key takeaways:

• Every OpenStack service has an exporter for Prometheus.
• Grafana visualizes API, instance, hypervisor, and volume metrics.
• The ELK Stack and Loki centralize logs from all nodes.
• Alertmanager turns metrics into proactive notifications.
• Watch RabbitMQ queues and OSD status as early-warning signals.
• A good alert is specific, measurable, and free of false positives.

In episode 19, we'll cover Troubleshooting & Operational Maintenance — using diagnostic commands like compute service list, network agent list, and volume service list, handling common issues such as failed instance spawns, down network agents, and auth errors, and day-2 operation strategies like Kolla-Ansible upgrades and RabbitMQ queue cleanup.