Learn Istio - Upgrade, Backup & Disaster Recovery

#Introduction

Installing Istio once is not enough — technology moves, and your mesh has to move with it. Episode 21 covers the three things that determine long-term survival: safe upgrades, backup and restore, and disaster recovery. It all comes down to one principle: how to get home safely when something does not go according to plan.

#A Safe Upgrade Path

#Upgrade Pre-requisites

Before raising the version, take the mandatory steps:

• Read the release notes of the target version and the list of changed features.
• Run istioctl x precheck on the cluster.
• Make sure the istioctl version matches the target version.
• Upgrade only one minor version per step.

istioctl x precheck
istioctl upgrade --dry-run --filename upgrade-manifest.yaml

istioctl x precheck checks compatibility before anything changes. --dry-run gives you a preview of the changes without applying them.

#Canary Control Plane and Workload Migration

The safest pattern: install the new version as a canary revision, then move workloads gradually:

istioctl install --revision=1-23-0 --set profile=default -y
kubectl label ns default istio.io/rev=1-23-0

The istio.io/rev=1-23-0 label makes new Pods use the new istiod version. Monitor for a few days, then migrate the remaining namespaces. Workloads still on the old revision keep being served by the old istiod — no downtime.

#Rollback Strategies

No upgrade is risk-free. Prepare the rollback from the start:

• Save the old istioctl version and manifests.
• Note the old revision before the upgrade begins.
• Roll back by returning the namespace revision label to the old version.

kubectl label ns default istio.io/rev=1-22-1 --overwrite

kubectl label ns default istio.io/rev=1-22-1 directs new workloads back to the old revision. Existing Pods are restarted gradually to return to the previous version. Remember: istioctl x upgrade does not migrate CRDs automatically — never downgrade CRDs to a version the old revision does not recognize.

#Backup and Restore of Configuration

#Backing Up Istio CRDs

Istio configuration is plain YAML — no expensive tooling is needed for a basic backup:

kubectl get $(kubectl api-resources --verbs=get -o name | grep istio.io) -A -o yaml > istio-config-backup.yaml

kubectl get ... -A -o yaml exports all resources from the Istio API groups into a single YAML file. Store it in separate storage (Git, S3, R2) so it is available when the cluster is lost.

#Full Backup with Velero

For full recovery including Namespaces and Secrets, use Velero:

velero backup create mesh-backup --include-namespaces istio-system,default --include-resources '*'
velero restore create --from-backup mesh-backup

velero backup create --include-namespaces istio-system,default stores all objects in those namespaces into object storage. Velero can also back up PersistentVolumes — important for Prometheus and Kiali.

#Cluster Recovery and Disaster Recovery

#The Recovery Sequence After a Disaster

When a new cluster is built from scratch, the recommended recovery order:

1. Install Kubernetes and basic tooling.
2. Install Istio with the same version and profile.
3. Restore the Istio CRDs from the YAML backup.
4. Restore Namespaces, Secrets, and applications (via GitOps or Velero).
5. Verify istioctl proxy-status shows everything SYNCED.
6. Verify metrics, traces, and the traffic flow from the chaos drill.

#A Tested DR Plan

Disaster recovery is not a document — it is practice. Schedule regular incident drills: build a cluster from scratch, restore the backup, and measure how long until all services serve traffic. This number (RTO) must match business expectations. Save the results as a runbook for episode 22.

#Summary

Episode 21 protected the mesh's journey: upgrades with a canary control plane and gradual workload migration, rollback strategies with revisions, CRD backups with kubectl or Velero, and a cluster recovery procedure tested through incident drills.

Key takeaways:

• Precheck and upgrade one minor version per step.
• A canary revision allows workload migration without downtime.
• Rollback uses the revision label to return to the old version.
• Back up Istio CRDs with a YAML export and Velero for full objects.
• Store backups in storage separate from the cluster.
• Restore order: Kubernetes, Istio, configuration, applications.
• A DR plan must be tested: measure RTO via regular incident drills.

In the final episode, episode 22, we will wrap everything up: production hardening and best practices — a security checklist with mTLS and least-privilege, an operational checklist with quotas and runbooks, and documentation standards for service owners.