Learn Velero - Backup Policies & Compliance

#Introduction

In episode 6 you learned about TTL — backups automatically deleted after their time. But there's a scenario TTL can't satisfy: regulations require data to be kept for years, and it must not be deletable — even by an admin with full access. That's where episode 15 comes in: turning Velero from a "neat backup tool" into a compliance-satisfying system.

The most fitting analogy is a notary's archive: important documents must not be tearable by anyone, including the person who created them. Object lock is the "filing cabinet locked from the inside" — once data goes in, no one can delete it until the lock period ends.

#Immutability with Object Lock

#The Concept

Object Lock (S3 / MinIO / GCS retention) makes objects in a bucket undeletable and unoverwritable for a certain period. The compliance level can be very strict: even the root account can't delete an object during the retention period.

Enable Object Lock when creating the bucket (S3 doesn't allow enabling it after the bucket exists):

Create a bucket with Object Lock

aws s3api create-bucket \
  --bucket velero-archive \
  --create-bucket-configuration LocationConstraint=ap-southeast-1 \
  --object-lock-enabled-for-bucket

#Connecting Velero to Object Lock

The BSL has an objectLock field that tells Velero backups in this location must not be deleted — Velero will refuse delete operations and warn when a TTL expires:

BSL with objectLock

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: archive
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: velero-archive
    prefix: backups
  config:
    region: ap-southeast-1
  objectLock: true
  accessMode: ReadWrite

Warning

objectLock: true is an explicit acknowledgment that backups in this location will never be deletable until the lock ends. Read it twice before enabling: a retention mistake (e.g. a 3-year lock on data that's no longer relevant) means objects pile up and add permanent storage costs.

#Separate BSL per Policy

#Why Separate Them

Different policies need different treatment:

Operational: TTL 30 days, deletable, low cost.
Compliance/archive: object lock, very long TTL or no TTL, archive cost (glacier/deep archive).

Mixing both in one bucket makes the policy unclear and hard to audit. The Velero solution: a separate BSL per policy, then direct backups to the right BSL.

Two BSLs with different policies

velero backup-location create ops \
  --provider aws --bucket velero-ops --config region=us-east-1 --default
 
velero backup-location create archive \
  --provider aws --bucket velero-archive \
  --config region=ap-southeast-1

#Schedules per Policy

Operational vs compliance schedules

velero schedule create daily-ops \
  --schedule="0 2 * * *" --include-namespaces prod --ttl 720h \
  --storage-location ops
 
velero schedule create monthly-archive \
  --schedule="0 3 1 * *" --include-namespaces prod --ttl 26280h \
  --storage-location archive

Monthly backups to the archive bucket with a 3-year TTL (26280 hours) and object lock at the bucket level — this is a common compliance pattern.

#Compliance Retention

#TTL as a Contract

Every backup has its own TTL; for backups that are a compliance obligation, make sure the TTL is consistent with the regulatory requirements — and recorded. Don't rely on memory:

Audit all backup TTLs

velero backup get

Backups with a shorter TTL than the regulatory requirement must be corrected immediately (recreate with the correct TTL) — because once the TTL expires, the objects are deleted automatically.

#Combining with Object Lock

The strongest pattern: long TTL + object lock + archive BSL. TTL is Velero's rule; object lock is the physical safety net in storage. If a bug deletes backups, object lock still holds them. If an object lock is wrong, the TTL at least limits how long the mistake lasts.

#Audit: Logs and Events

#Velero Server Logs

All operations are recorded in the server pod logs:

Stream server logs

kubectl logs -n velero deploy/velero --tail=200

For long-term audit, ship the logs to an observability system (Loki/ELK/CloudWatch). Use the log configuration in velero install if needed, or a sidecar on the velero deployment.

#Backup/Restore Events

Kubernetes events record status changes for every operation:

Grep for the important phases: Started, Completed, Failed, Expired. These events can feed a compliance dashboard like "required-retention backups completed / failed".

Tip

Build an automatic compliance report: a cron/CI job that runs velero backup get, filters compliance backups, and raises an alert when any is failed or its TTL deviates from the rules. A printed report is far more valuable when an auditor asks "show me the evidence".

#Closing

Key takeaways:

Object lock (S3/MinIO/GCS) makes backups undeletable — enable it when creating the bucket.
A BSL with objectLock: true tells Velero not to delete backups in that location.
Separate BSLs per policy: operational (short TTL) vs compliance archive (lock + long TTL).
TTL is the rule; object lock is the safety net — combine both.
Audit via server logs (kubectl logs) and events (kubectl get events), plus automatic reports.

In episode 16 next, we face the bitter reality every operator knows: Troubleshooting & Debugging — velero backup describe --details, velero backup logs, and velero bug, plus common cases like invalid BSL, failed snapshots, node-agent crashes, and pending kopia backups.