Learn Velero - Backup Hooks: Pre/Post (Ensuring Consistency)

#Introduction

In episode 8 you learned how to store volume data. But there's a deeper question: is the stored data consistent? Imagine photocopying a book while the author is rearranging it — page 5 is already copied, but the author will change its contents right after that page. The copy is "successful", but the returned book is incomplete.

The same problem occurs when backing up a database or stateful application: files are being written continuously, and a snapshot taken mid-write produces corrupt data that can't be opened. The solution is backup hooks — commands Velero runs inside the pod right before and after the volume data is copied.

#The Concept of Backup Hooks

#Pre and Post Hooks

• Pre hook: run before the volume backup starts. Its job is to prepare a consistent state — flushing buffers, pausing writes, or creating a dump.
• Post hook: run after the backup finishes. Its job is to restore normal operation — resuming writes, taking the app out of maintenance mode.

Both are executed inside a specific container of the pod being backed up, using commands you define.

#Configuration via Annotations

Hooks are added as annotations on the pod (not the deployment — the pod is the unit that gets backed up):

Hook annotations on a pod

annotations:
  pre.hook.backup.velero.io/container: db
  pre.hook.backup.velero.io/command: '["/bin/sh", "-c", "pg_ctlcluster 16 main stop --mode=fast"]'
  pre.hook.backup.velero.io/on-error: Fail
  pre.hook.backup.velero.io/timeout: 120s
  post.hook.backup.velero.io/container: db
  post.hook.backup.velero.io/command: '["/bin/sh", "-c", "pg_ctlcluster 16 main start"]'
  post.hook.backup.velero.io/timeout: 120s

The important part of each annotation:

• pre.hook.backup.velero.io/command — the JSON array command to run.
• pre.hook.backup.velero.io/container — the target container (default: first container).
• pre.hook.backup.velero.io/timeout — time limit (default 30s).
• pre.hook.backup.velero.io/on-error — Fail (fail the backup) or Continue.

#Exec Hooks vs Init Hooks

Besides exec (running commands in a running container), Velero has init hooks via the initContainers.velero.io/* annotations — injecting an init container before backup. This is useful for commands that need special tooling (e.g. a pg_dump that isn't in the application image).

#Case Study: PostgreSQL

#The Problem

An EBS snapshot or copying PostgreSQL data files mid-transaction produces a corrupt database — the wal (write-ahead log) is out of sync with the data files. File-level backup makes it worse because it copies files one by one. For a usable backup, the database must be in a "quiesced" state.

#Approach 1: Stop/Start

The simplest and safest approach: stop the server, back up, start it again.

PostgreSQL stop/start hooks

annotations:
  pre.hook.backup.velero.io/container: db
  pre.hook.backup.velero.io/command: '["/bin/sh", "-c", "pg_ctlcluster 16 main stop --mode=fast && pg_dumpall > /backup/dump.sql"]'
  pre.hook.backup.velero.io/on-error: Fail
  pre.hook.backup.velero.io/timeout: 300s
  post.hook.backup.velero.io/container: db
  post.hook.backup.velero.io/command: '["/bin/sh", "-c", "pg_ctlcluster 16 main start"]'
  post.hook.backup.velero.io/timeout: 300s

velero backup create pg-backup --include-namespaces app --default-volumes-to-fs-backup

Note: velero executes the pre hook before copying the volume, so the order is correct — DB stops, volume is copied, DB starts again.

#Approach 2: pg_dump (Logical Backup)

For RPOs that don't allow downtime, use a logical dump as an additional artifact:

kubectl exec -n app deploy/db -- pg_dump -U postgres -Fc -f /backups/pg.dump

Combine them: the file-level backup copies the data files, and the logical dump provides a more flexible point-in-time restore option. Many teams use both approaches at once.

#Case Study: Other Stateful Applications

#MySQL

annotations:
  pre.hook.backup.velero.io/container: mysql
  pre.hook.backup.velero.io/command: '["/bin/sh", "-c", "mysql -uroot -p$MYSQL_ROOT_PASSWORD -e \"FLUSH TABLES WITH READ LOCK;\\nFLUSH LOGS;\\nUNLOCK TABLES;\""]'
  pre.hook.backup.velero.io/on-error: Fail
  pre.hook.backup.velero.io/timeout: 120s

FLUSH TABLES WITH READ LOCK locks all tables and makes a file-level backup consistent. More elegant production alternatives are mysqldump or xtrabackup.

#Redis / Cache / Queue

For workloads that buffer data in memory, make sure persistence is enabled (e.g. Redis BGSAVE or appendonly) and back up the .rdb/.aof files — or simply accept a small RPO since the data can be rebuilt.

#Common Pitfalls

• Wrong container: a hook targets the db container, but the pod has a differently named container → hook fails, backup fails (on-error: Fail). Check with kubectl get pod -o jsonpath='{.spec.containers[*].name}'.
• Timeout too small: pg_dump of a large database can exceed the 30-second default. Increase the timeout.
• Hooks on the Deployment, not the Pod: annotations must be on the pod template (spec.template.metadata.annotations), not at the Deployment level.
• In-memory state: applications storing state only in RAM (no persistence) won't survive any snapshot — hooks can't save a poorly designed architecture.

#Closing

Key takeaways:

• A volume backup without consistency = corrupt data "successfully" stored.
• Pre hooks prepare state; post hooks restore normal operation.
• Configure via the pre.hook.backup.velero.io/* and post.hook.backup.velero.io/* annotations on the pod.
• Set timeout and on-error (Fail/Continue) for clear behavior.
• PostgreSQL/MySQL need quiesce (stop/start, flush, or logical dump); Redis needs active persistence.

In episode 10 next, we assemble the restore workflow in full: full restore to a new cluster, --namespace-mappings for production→staging, and troubleshooting failed restores like pending PVCs and different storage classes with --storage-class-mappings.