Learn Proxmox VE - Troubleshooting & Disaster Recovery

#Introduction

Everything you've built since episode 0 will have problems at some point. The difference between a great administrator and an average one isn't the ability to avoid problems — it's the speed and calmness with which they solve them. Episode 19 trains your diagnostic instincts.

We'll master the diagnostic commands, read logs properly, solve the most common problems that occur in Proxmox, and then put together a disaster recovery procedure you can rely on when it's truly needed.

#Proxmox Diagnostic Commands

#Cluster and HA Status

When there's a problem, start with the most general status. The following commands give you a picture of system health:

pvecm status
ha-manager status
qm list
pct list

pvecm status shows cluster health and quorum, ha-manager status shows HA resources, while qm list and pct list show all VMs and containers along with their status.

#Verifying Storage and ZFS

Storage is often the culprit:

pvesm status
zpool status
df -h

zpool status shows ZFS pool health — pay attention to DEGRADED or FAULTED statuses on disks.

#Log Analysis

#journalctl for Proxmox Services

Logs are the best witnesses. The main Proxmox services — pvedaemon, pveproxy, pve-cluster — write logs to the systemd journal:

journalctl -u pvedaemon -u pveproxy -b

The journalctl -u pvedaemon -u pveproxy -b command shows the logs of two main services since the last boot. Add the -f flag to follow the log in real-time while diagnosing a live problem.

#Task Logs and QEMU

Every Proxmox operation leaves a trace in the task log. For VM start failures, look at the task log and the configuration file:

qm start 100 --debug
cat /etc/pve/qemu-server/100.conf

The qm start 100 --debug command shows complete details when a VM fails to start — including the QEMU error, which is the main clue.

#Troubleshooting Common Problems

#Cluster Not Quorate or Split-Brain

If pvecm status shows QUORUM is not reached, it means the vote count is below the majority. Common causes: a dead node or a broken Corosync network. Check connectivity between nodes and make sure the node count is odd. Never power on two nodes at the same time in a small cluster.

#VM Won't Start (QEMU Errors)

A VM that fails to start usually leaves a specific QEMU message. Common causes: no disk space, the disk is unavailable on the node, or a lock left over from a previous operation. Remove the lock with qm unlock 100 if needed.

#LXC Permission Issues

A container that won't start is often caused by a changed storage configuration or an inconsistent template. Check the pct start log and make sure the container rootfs is available on the configured storage.

#Storage Full and ZFS Pool Degraded

Full storage causes problems for every VM on it. Delete unnecessary files, add capacity, or move data. For a DEGRADED ZFS, identify the failed disk with zpool status, replace it, and let ZFS perform the resilver.

#Node Network Unreachable

A node that can't be reached from the network usually has a problem in the network configuration. Access it via the physical console or IPMI, check /etc/network/interfaces, and test with ip a.

#Disaster Recovery Plan

#Recovering a VM from Backup

When a VM is badly damaged, the recovery procedure from backup is your lifeline:

1. Make sure the backup storage is still available.
2. Restore the VM with qmrestore.
3. Start the VM and verify the services.
4. Document the recovery time for evaluation.

qmrestore /var/lib/vz/dump/vzdump-qemu-100-2026_08_10-02_00_00.vma.zst 100
qm start 100

#Recovering a Failed Node

If a node truly fails (dead hardware), the procedure is:

• Make sure the VM data is available via replication or backup.
• On a healthy node, remove the node from the cluster with pvecm delnode.
• Replace the hardware, reinstall Proxmox, and join it back to the cluster.
• Restore the backed-up VMs to the new node.

#Closing

Episode 19 trained you to face failures: mastering the diagnostic commands, analyzing logs, solving common problems like split-brain and full storage, and putting together a tested disaster recovery procedure.

The key takeaways:

• Diagnosis starts with pvecm status, ha-manager status, qm list, and pct list.
• Logs are in journalctl: pvedaemon and pveproxy for the main services.
• Split-brain happens when quorum is lost; keep the node count odd.
• VM start failures are often due to a full disk or leftover locks.
• A DEGRADED ZFS needs a disk replacement and resilver.
• Test disaster recovery procedures periodically.

In the next episode, episode 20, we will cover a production-grade Proxmox infrastructure case study — designing a complete architecture from a 3-node cluster, storage tiers, networking, automation, to observability, plus a production readiness checklist and maintenance routine. This is the peak of your journey; let's assemble all your skills into one complete design!