Learn OpenTofu - Disaster Recovery (DR) & Encrypted State Recovery

#Introduction

In the previous episode 18 we covered drift detection and automatic remediation, ensuring infrastructure always returns to its declared state. But there's a scenario more frightening than drift: a state file that can't be read at all. Encrypted state that loses its key is no longer a question of recovery speed — it's a question of whether recovery is even possible.

In this episode 19, we'll discuss Disaster Recovery & Encrypted State Recovery. We'll walk through recovery procedures for encrypted state when the KMS key or passphrase has problems, and design backup and versioning strategies using S3 and GCS. This is the episode about peace of mind: ensuring the worst case never becomes the end of everything.

#Why State Is Your Most Valuable Asset

In episode 5 we learned that state is the "inventory" connecting HCL code to real cloud resources. Without state, OpenTofu no longer knows which resources exist — and a careless tofu apply could have clusters, buckets, or databases recreated from scratch, or even destroyed.

With the native client-side encryption feature covered in episode 6, state is stored as ciphertext. This is very secure against theft, but it brings a consequence: state security now depends on the ability to decrypt. A lost key is the same as lost data.

#Scenario 1: The KMS Key Has Problems

Suppose we encrypt state using AWS KMS:

terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "arn:aws:kms:ap-southeast-1:123456789012:key/abc123"
    }
 
    method "aes_gcm" "main" {
      keys = key_provider.aws_kms.main
    }
 
    state {
      method = method.aes_gcm.main
    }
  }
}

A KMS key can have problems for various reasons: accidentally deleted, disabled, moved regions, or an IAM role that no longer has decrypt permission. The symptom is usually an error when running tofu plan or tofu state pull.

Recovery steps:

1. Diagnose — check the key's status in the KMS console: is it enabled, disabled, or pending deletion.
2. Restore the key — re-enable it or cancel the deletion from KMS. Make sure the IAM role has kms:Decrypt permission.
3. Verify the region — make sure kms_key_id points to the correct region and account.
4. Test access — run tofu state pull to confirm the state can be decrypted.
5. Verify the plan — run tofu plan and make sure there are no unexpected changes, indicating the state reads perfectly.

#Scenario 2: The PBKDF2 Passphrase Is Lost

For simpler environments, we might use the pbkdf2 key provider with a passphrase:

terraform {
  encryption {
    key_provider "pbkdf2" "main" {
      passphrase = var.state_passphrase
    }
 
    method "aes_gcm" "main" {
      keys = key_provider.pbkdf2.main
    }
 
    state {
      method = method.aes_gcm.main
    }
  }
}

The passphrase can be read from the TF_OPENTOFU_STATE_PASSPHRASE environment variable so it never appears in code. Problems arise when the passphrase is lost or mistyped — and since key derivation is deterministic, a single wrong character makes the entire state unreadable.

To minimize this risk, store the passphrase in a secret manager (Vault, AWS Secrets Manager, or GCP Secret Manager), keep a backup copy in a separate location, and test recovery periodically. The rule of thumb: if no second person can reconstruct the passphrase, it isn't a backup.

#Gradual Recovery Procedure

When an incident occurs, follow this procedure in order — don't jump straight to reconstruction:

1. Stay calm and don't apply — a tofu apply command with a problematic state can make things worse.
2. Try decrypting with the correct key — fix permissions, region, or passphrase as in the scenarios above.
3. Use versioning — if the old key is truly lost, pull a previous state version from the S3/GCS backup that's still decryptable with the old key, then re-encrypt.
4. Use fallback for rotation — OpenTofu provides a fallback mechanism so a new key can read state written with the old key.
5. Document and evaluate — record the root cause and strengthen the procedure so it doesn't repeat.

#Key Rotation with Fallback

OpenTofu supports key rotation without losing access to old state via the fallback attribute. When the old key has problems, we introduce a new key as the main method and keep the old key as the fallback:

terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "arn:aws:kms:ap-southeast-1:123456789012:key/new-key"
    }
 
    key_provider "aws_kms" "old" {
      kms_key_id = "arn:aws:kms:ap-southeast-1:123456789012:key/old-key"
    }
 
    method "aes_gcm" "main" { keys = key_provider.aws_kms.main }
    method "aes_gcm" "old"  { keys = key_provider.aws_kms.old }
 
    state {
      method   = method.aes_gcm.main
      fallback = method.aes_gcm.old
    }
  }
}

With this configuration, OpenTofu writes new state with the new key but can still read old state as long as the old key is available as a fallback. Once all state is re-encrypted with the new key, the fallback can be removed. This is how you rotate keys without downtime — like changing a door lock without locking the residents out.

#Backup & Versioning Strategies

Recovery can never succeed without proper backups. The two main strategies used in the real world:

#Versioning on AWS S3

The bucket storing state must enable versioning — so every state version (including deleted ones) stays stored:

resource "aws_s3_bucket" "state" {
  bucket = "org-opentofu-state"
}
 
resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration {
    status = "Enabled"
  }
}
 
resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

A versioned bucket becomes a time machine for state. If the latest state is corrupted or unreadable, we can retrieve a previous version.

#Versioning on GCP GCS

The same approach applies on GCP using google_storage_bucket with versioning:

resource "google_storage_bucket" "state" {
  name     = "org-opentofu-state"
  location = "ASIA-SOUTHEAST2"
 
  versioning {
    enabled = true
  }
}

#Testing Recovery

A backup that's never tested isn't a backup. Schedule periodic recovery drills — quarterly, for instance — with these steps:

1. Take an old state version from the versioned bucket.
2. Restore it to the staging environment.
3. Verify tofu plan runs without errors.
4. Record the recovery time and evaluate it against the RTO target.

These drills ensure all the procedures above actually work when needed, not just look tidy on paper.

#Conclusion

In episode 19, we discussed Disaster Recovery & Encrypted State Recovery. We learned recovery procedures when the KMS key or passphrase has problems, key rotation techniques with fallback, and backup and versioning strategies on S3 and GCS.

Key takeaways:

• Encrypted state is only as safe as its key; a lost key means lost data.
• Diagnose KMS: check the key's status, region, and kms:Decrypt permission.
• PBKDF2 passphrases are deterministic — one wrong character makes state unreadable.
• tofu state pull is a quick verification tool for testing decryption ability.
• The fallback attribute enables key rotation without downtime.
• Enable versioning on S3 and GCS as a time machine for state.
• A backup without a recovery test isn't a backup.

Disaster recovery is proof that your systems withstand the worst case. In the next episode, episode 20 — the final episode — we'll combine all the capabilities learned from episode 0 through 19 into one Complete Production-Grade Enterprise OpenTofu Architecture. See you there!