Learn OpenStack - Troubleshooting & Operational Maintenance

#Introduction

All great architecture eventually meets one reality: something will break. Instances refuse to be born, networks jam, volumes fail to attach, logins are rejected. What distinguishes a professional operator isn't the ability to avoid problems, but the way they diagnose and recover quickly and calmly.

Episode 19 is the most practical episode in this series: you'll memorize the core diagnostic commands, learn troubleshooting patterns for the four most common problems — failed instance spawns, network issues, volume attach failures, and authentication errors — then master day-2 operations like upgrades and queue cleanup.

#OpenStack Diagnostic Commands

#Assessing Service Health

Before diving into logs, check the service state from one place:

openstack compute service list
openstack network agent list
openstack volume service list

The output of openstack compute service list shows the status of all Nova services per node. Watch the State column (must be up) and the Status column (must be enabled). openstack network agent list and openstack volume service list do the same for Neutron and Cinder. These three are the first line of every diagnosis.

#Checking Supporting Infrastructure

When services look normal but problems persist, look at the supporting components:

sudo rabbitmqctl status
mysql -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

rabbitmqctl status confirms the message broker is alive, and SHOW STATUS LIKE 'wsrep_cluster_size' verifies all Galera nodes are in one cluster. A full or dead memcached is also a frequent cause of weird errors, since Keystone tokens are stored there.

#Troubleshooting Common Issues

#Instance Failed to Spawn

Symptom: openstack server create results in ERROR status. Diagnosis order:

openstack server show my-server -f value -c fault
openstack compute service list -f value -c host -c state

The output of openstack server show -c fault shows the reason from the scheduler or compute. The most common causes: insufficient resources (RAM/vCPU quota), a corrupted image, or a down compute service. Check the nova-compute log and make sure the Placement service reports the correct resources:

openstack resource provider list

#Networking Issues

Symptom: an instance without an IP, or DHCP not working. Check the agents and metadata:

openstack network agent list -f value -c agent_type -c host -c alive
openstack network agent list --dhcp

The output of openstack network agent list shows whether the dhcp, metadata, and l3 agents are still alive. A dead agent means DHCP and the metadata service aren't serving new instances. Restart the agent from the relevant node and verify the namespaces:

sudo systemctl restart neutron-dhcp-agent

#Volume Attach Failures

Symptom: openstack server add volume hangs or errors. Check the Cinder backend status and the connection to Ceph:

openstack volume service list
ceph -s

The output of openstack volume service list shows the backend state; ceph -s confirms the cluster is healthy. Common causes: the pool is out of space, or the compute node can't reach Ceph. Also make sure the keyring and Ceph configuration are the same on all nodes using volumes.

#Keystone Auth 401/403

Symptom: the openstack CLI refuses to log in. The three most common causes: expired token (re-source openrc), wrong credentials/project/domain, or unsynchronized clocks between nodes. Check time synchronization first:

chronyc tracking | grep -E "Stratum|Offset"

The output of chronyc tracking shows the node's time offset. If the offset is large, Fernet tokens are considered invalid on other nodes — synchronize all nodes with chrony/NTP as the first step in every auth debug.

#Day-2 Operations

#Upgrade Strategy

Upgrading production OpenStack isn't guesswork — it's a documented playbook:

sed -i 's/openstack_release: "caracal"/openstack_release: "dalmatian"/' /etc/kolla/globals.yml
kolla-ansible -i multinode prechecks
kolla-ansible -i multinode upgrade

kolla-ansible -i multinode upgrade raises services one by one with rolling restarts. The principles: always run prechecks first, do it in a maintenance window, and make sure the database is backed up before upgrading. Don't skip major versions without reading the release notes.

#Database Maintenance

The database is the cluster's source of truth — treat it like treasure:

mysql -e "OPTIMIZE TABLE nova.instances;"
mysqldump --all-databases > /backup/openstack-2026-08-10.sql

OPTIMIZE TABLE reorganizes tables, and mysqldump creates a restorable backup. Scheduled backups and database size monitoring are routines you can't skip.

#RabbitMQ Queue Cleanup

Accumulated queues (swollen unacked messages) slow down every service:

sudo rabbitmqctl list_queues name messages messages_unacknowledged
sudo rabbitmqctl reset

rabbitmqctl list_queues reveals the problematic queues. If a queue is corrupted and unrecoverable, rabbitmqctl reset returns the broker to a clean state — but this sacrifices queue state, so make sure it's the last resort after other options fail.

#Summary

Episode 19 equips you with real operational skills: mastering the compute service list, network agent list, and volume service list diagnostic commands, handling common issues like failed instance spawns, down networks, failed volume attaches, and auth errors, and running day-2 operations like Kolla-Ansible upgrades, database maintenance, and RabbitMQ queue cleanup.

Key takeaways:

• Diagnose starting from service status, then logs, then infrastructure.
• Instance ERROR: check the fault, resources, and compute services.
• A dead agent means DHCP and metadata aren't being served.
• Auth 401/403: check time synchronization between nodes first.
• Upgrades require prechecks and backups before execution.
• RabbitMQ queues and the database need regular maintenance.

In episode 20 — the final episode — we'll cover the Complete Production-Grade Private Cloud Architecture Case Study — designing a full enterprise architecture from the Kolla-Ansible deployment engine, the Nova compute layer with Ceph, Neutron networking with VLAN, VXLAN, DVR, and Octavia, Ceph unified storage with Barbican, to observability and security, ending with a production readiness checklist and maintenance routine.