Learn Velero - Troubleshooting & Debugging

#Introduction

So far everything has gone smoothly. Time to be honest: in production, backups will fail — credential rotation, full nodes, new plugin versions, wrong network policies. Episode 16 trains your instinct to stay calm and solve problems systematically, not by random trial and error.

Think like a mechanic: no one diagnoses a car by dismantling every part at once. There's an order — check symptoms, read the data, isolate the cause, then fix. Velero gives you the right instruments: describe, logs, and bug. Let's learn how to use them.

#Basic Diagnosis Tools

#velero backup describe --details

The first command in every incident:

velero backup describe my-backup --details

What to look for here: phase (Completed, Failed, PartiallyFailed), the number of succeeded/failed resources, and the list of backed-up volumes. PartiallyFailed usually means some volumes failed — read further with logs.

#velero backup logs

The logs give you the actual error messages:

velero backup logs my-backup | grep -i error | head -20

Or for restores:

velero restore logs my-restore | grep -i error | head -20

#velero bug

For recurring issues or something suspicious of being a bug, gather information to report on GitHub:

velero bug

This command prints the client version, cluster status, and opens an issue template — making sure your report contains actionable data for maintainers.

#Common Case 1: Invalid / Unavailable BSL

Symptom: velero backup-location get shows Unavailable; backups fail with a bucket connection error.

Diagnosis:

velero backup-location get
kubectl logs -n velero deploy/velero | grep -i "backupstorage"

Causes and solutions:

• Wrong/rotated credentials → update the Secret (episode 13) and restart the pod.
• Wrong region or unreachable endpoint → check region, s3Url, and NetworkPolicy (episode 14).
• Bucket doesn't exist → create the bucket; the BSL doesn't create buckets automatically.
• TLS mismatch (internal CA) → set caCertRef on the BSL.

#Common Case 2: No Snapshots (Wrong VSL)

Symptom: backup shows Completed but not a single volume has data — manifests exist, data is missing. In describe --details, the volume section is empty or marked "skipped".

Diagnosis:

velero snapshot-location get
kubectl logs -n velero deploy/velero | grep -i snapshot

Causes and solutions:

• VSL not configured / wrong region → create the correct VSL (velero snapshot-location create ...).
• Snapshotter plugin not installed → velero plugin add velero/velero-plugin-for-aws:v1.14.0.
• Cluster uses CSI and the CSI plugin is missing → see episode 19.
• No node-agent and the backup was run without --default-volumes-to-fs-backup → PVCs are only backed up as manifests. This is the most common case in installations without cloud snapshots: volume data is silently not stored.

#Common Case 3: Node-Agent Crash

Symptom: the node-agent DaemonSet keeps restarting (CrashLoopBackOff) on some nodes; file-level backups fail on those nodes.

Diagnosis:

kubectl get pods -n velero -l name=node-agent
kubectl logs -n velero -l name=node-agent --tail=50

Causes and solutions:

• Image version doesn't match the server → match the node-agent image version to the Velero version.
• Resource limits too small → raise node-agent CPU/memory limits.
• HostFilesystem/volume mount unavailable (e.g. different kubelet directory) → set --kubelet-root-dir at install.
• ReadOnlyRootFileSystem: true blocks the kopia cache → give write access to a volume on the cache directory (mentioned in the 1.18 docs).

#Common Case 4: Pending Kopia Backups

Symptom: file-level backup stays InProgress for a very long time, or the PodVolumeBackup hangs.

Diagnosis:

Inspect PodVolumeBackups

kubectl get podvolumebackups -n velero
kubectl describe podvolumebackups <name> -n velero
kubectl logs -n velero <node-agent-pod> | grep -i kopia

Causes and solutions:

• Large volume + low parallelism → raise --parallel-files-upload in config (episode 20).
• Node full → free up node disk space.
• Repository key missing / repository corrupt → check the kopia-repo-* Secret; don't delete it.
• Small CPU limit: Velero 1.18 uses Golang 1.25, which respects pod CPU limits — throughput can drop sharply if the limit is lowered. Adjust parallel-files-upload to match the available CPU limit.

#A Structured Diagnosis Method

1. Gather status: velero backup get, backup-location get, snapshot-location get.
2. Zoom in with describe: velero backup describe <name> --details.
3. Read the actual errors: velero backup logs <name> | grep -i error.
4. Check the suspected components: server pod (kubectl logs deploy/velero), node-agent.
5. Fix, rerun the backup, verify the restore.

#Closing

Key takeaways:

• velero backup describe --details → velero backup logs → velero bug is the diagnosis flow.
• BSL Unavailable: credentials, region, endpoint, TLS — check in that order.
• A "Completed" backup without volume data = the biggest danger; do routine restore verification.
• Node-agent crash: match image versions, check resources, check the kubelet root dir, allow the kopia cache.
• Pending kopia backups: raise parallelism, check node disk, never delete the repository Secret.

In episode 17 next, we dissect the version you're using: Velero 1.18 & the Latest Features — Kubernetes compatibility, kopia as the default uploader, the storage-class-mappings improvements, and the 1.14 to 1.18 release history.