Learn Terraform - Disaster Recovery (DR) & State Recovery

#Introduction

After discussing infrastructure drift and refactoring in episode 18 — keeping code, state, and cloud aligned — in this episode we'll face the most tense scenario in the entire IaC journey: a corrupted or lost state file.

Imagine this. Thursday night, 11:40 PM. There's a production incident, and you need to fix the infrastructure immediately. You open the terminal, run terraform init, then terraform plan. What appears isn't a change plan — but a single line that makes your heart stop for a moment:

Error: Unmarshal tfstate: failed to unmarshal state file
  json: cannot unmarshal object into Go struct field State.terraform_version of type string
 
Terraform state file is invalid and cannot be read. If this is unexpected,
please contact your support provider for assistance.

Or worse: terraform init succeeds, but terraform plan shows Plan: 78 to add — even though the infrastructure already exists. That means the state file is lost or empty, and Terraform thinks there's nothing it's managing. All your production resources are considered "new". One naive terraform apply would create duplicates or destroy everything.

Why is this topic important? Because in SRE/DevOps work, "how fast you recover" determines a team's reputation more than "how well you write code". The state file is Terraform's long-term memory — losing it doesn't destroy the infrastructure, but it makes it unmanageable. In this episode you'll learn to build the right disaster recovery foundation, restore state from backups, perform terraform state pull and terraform state push operations safely, all the way to manual re-import when state can no longer be saved.

#Main Discussion

#Why State Backup Is the DR Foundation

Since episode 5, we've known that the state file contains the mapping between resource blocks in the code and the real object IDs in the cloud. Inside it's recorded what Terraform owns: which bucket, which instance, which database — complete with all attributes and secret keys.

The logical consequence: if state is lost, Terraform is blind. It still knows what the code wants, but not what already exists in the cloud. The results:

• terraform plan shows all resources as "to be created" (because there's no ownership record).
• terraform apply would create duplicate resources (because it doesn't know the old resources still exist).
• terraform destroy is even more dangerous — without state, it might think nothing needs to be destroyed (that's the good news), but if state is partially corrupted, it could destroy the wrong resources.

A fitting analogy: the state file is like a vehicle ownership certificate (BPKB). The car is still in the garage, still usable — but without the certificate, you can't prove ownership, can't sell it, and someday might fight over it with someone else who claims to own it.

#S3 Backend Versioning + DynamoDB Locking: The DR Foundation

The good news is, the DR foundation was already built in episode 5: a remote state backend on S3 with versioning, encryption, and DynamoDB for locking. Let's see why all three together form a layered defense:

Here's the backend configuration that serves as the DR foundation in this series:

terraform {
  backend "s3" {
    bucket         = "myapp-tfstate-bucket"
    key            = "production/infrastructure/terraform.tfstate"
    region         = "ap-southeast-1"
    encrypt        = true
    dynamodb_table = "myapp-tfstate-lock"
  }
}

But wait — there's something easy to miss: the S3 backend doesn't automatically enable versioning. Versioning is a property of the bucket itself, and must be enabled once — ideally with Terraform (using a temporary local backend or a separate state) or via the console when first creating the bucket:

resource "aws_s3_bucket" "tfstate" {
  bucket = "myapp-tfstate-bucket"
 
  lifecycle {
    prevent_destroy = true
  }
}
 
resource "aws_s3_bucket_versioning" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
 
  versioning_configuration {
    status = "Enabled"
  }
}
 
resource "aws_s3_bucket_server_side_encryption_configuration" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
 
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}
 
resource "aws_s3_bucket_public_access_block" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
 
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

With versioning enabled, every time terraform apply writes new state, the old version stays stored. If state is ever taken out (for example due to a wrong manual operation or corruption), you just restore the previous version — exactly like restoring a file from the recycle bin, but with a complete historical trace.

#Scenario: Corrupted State File

State corruption can come from many directions — a JSON file damaged during upload, incomplete truncation by a wrong manual operation, a human editing JSON (breaking the episode 8 rule!), or a bug in a particular Terraform version. The symptoms vary:

• Error: Unmarshal tfstate: failed to unmarshal state file — invalid JSON.
• Error: Missing resource instance key or odd address errors — partially broken structure.
• Plan: N to add even though the infrastructure already exists — empty/overwritten state.
• All resources appear changed (Plan: N to change) for no reason — corrupted attributes.

The diagnosis sequence is always the same:

terraform state list          # Can the resource list still be read?
terraform state show <addr>   # Do the attributes of a specific resource make sense?

If state list fails or the results don't make sense, don't proceed to apply. Stop. The next step is recovery — and recovery starts with understanding the two rescue commands: terraform state pull and terraform state push.

#terraform state pull: Fetching & Printing Raw State

terraform state pull fetches state from the backend and prints it to stdout in raw JSON format — exactly what's stored on S3. This is how you "open the state engine" for inspection, backup, or analysis:

terraform state pull

The output is a large JSON document starting roughly like this (truncated for readability):

{
  "version": 4,
  "terraform_version": "1.9.0",
  "serial": 42,
  "lineage": "7b1e2c3d-4a5b-6c7d-8e9f-0a1b2c3d4e5f",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "aws_instance",
      "name": "web",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "ami": "ami-0abcdef1234567890",
            "id": "i-0abcd1234efgh5678",
            "instance_type": "t3.medium",
            ...
          }
        }
      ]
    }
  ]
}

Note the two key fields at the top:

• serial — the state revision sequence number. The S3 backend (and most remote backends) refuses overwrites with a state whose serial is lower than the stored one. This is a built-in safeguard so old state doesn't accidentally overwrite new state.
• lineage — the state's unique "fingerprint". Two states from the same project must have the same lineage; if they differ, they come from different beginnings (a sign of two separate states that might conflict).

The most common uses of state pull in a DR context:

1. Quick backup — save a state snapshot before a risky operation.
2. Forensic inspection — look at the state's contents when diagnosing corruption.
3. Moving between backends — pull from one backend, push to another.

Save the pull result to a file:

terraform state pull > backup-tfstate-$(date +%F).json
ls -la backup-tfstate-*.json

#terraform state push: Overwriting State — With Full Caution

terraform state push is the reverse: takes state from stdin or a file, then overwrites it onto the backend. This is a very dangerous operation — without sufficient safeguards, one wrong push could overwrite healthy state with old or corrupted state.

terraform state push backup-tfstate-2026-08-01.json

When is state push legitimately used?

• Restoring state to a known-healthy version after corruption.
• Migrating state between backends (together with pull).
• Restoring state after a validated manual repair.

And when never:

• When another execution is currently running (locking doesn't fully apply to manual pushes).
• When you just want to "change a small number" — use the official subcommands (state mv, state rm, etc.).
• When you haven't backed up the current state yet.

terraform state pull > state-backup.json
python3 -m json.tool state-backup.json   # validate the JSON
jq '.resources | length' state-backup.json

terraform state push state-backup.json
terraform state list                     # verify state reads back correctly

#Repairing State JSON Manually (With a Backup First)

Sometimes corruption is partial — the JSON structure is mostly intact, with only one or two resources damaged. In that case, manual repair is possible, but only with a strict procedure. Remember the rule from episode 8: never edit state manually — that rule prevents bad habits. For DR rescue, there's a controlled exception, and here are the steps:

1. Lock first — make sure no other execution is running (locking active, or coordinate with the team).
2. Double backup — terraform state pull > backup.json, also copy the latest S3 version.
3. Repair on a copy — never edit directly on the backend.
4. Validate the JSON — make sure the file is still valid JSON and the state structure makes sense.
5. Push & verify — state push, then state list and plan to ensure no resource appears "lost" or "new".

# 1. Back up the state to be repaired
terraform state pull > state-broken.json
 
# 2. Copy for editing — do NOT edit the original file
cp state-broken.json state-fixed.json
 
# 3. Edit state-fixed.json with an editor (only the genuinely broken parts)
 
# 4. Validate the JSON structure
python3 -m json.tool state-fixed.json > /dev/null && echo "JSON valid"
 
# 5. Push the repair result
terraform state push state-fixed.json
 
# 6. Final verification
terraform state list
terraform plan -detailed-exitcode

#Recovery When State Is Totally Lost: Re-import with import Blocks

The worst-case scenario: there's no backup (a bucket without versioning, or even a deleted bucket), and state is truly empty. Fortunately, there's one last rescue path: rebuilding state with import — exactly the technique from episode 8.

The principle: the resources in the cloud are still alive; we just need to tell Terraform who they are. Since Terraform 1.5+, the cleanest way is the declarative import block — we declare the address of every existing resource:

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.medium"
}
 
resource "aws_db_instance" "primary" {
  identifier     = "myapp-postgres"
  engine         = "postgres"
  instance_class = "db.r6g.large"
}
 
import {
  to = aws_instance.web
  id = "i-0abcd1234efgh5678"
}
 
import {
  to = aws_db_instance.primary
  id = "myapp-postgres"
}

The full recovery flow from zero:

terraform init                              # 1. Initialize the empty backend
terraform plan                              # 2. Verify "Plan: N to import"
terraform apply                             # 3. Execute import → state rebuilt
terraform plan -detailed-exitcode           # 4. Verify: exit 0 = no drift

#DR Runbook Checklist

This checklist is a runbook that must be practiced — not just read. Keep it as a team document, and drill it (tabletop exercise) at least quarterly:

#DR & State Recovery Best Practices

• Never edit state while someone else is running. Always coordinate (or let the DynamoDB lock hold it) — two simultaneous writers is recipe number one for corruption.
• Always back up before manual operations. Every time you're about to state push, do a mass state mv, or edit JSON, run terraform state pull > backup.json first. One cheap command that saves the whole team.
• Use official subcommands rather than JSON edits. state mv, state rm, state replace-provider are always safer than touching raw JSON.
• Reduce state writers. The more places that can write state (laptops, pipelines, manual tools), the bigger the chance of conflict. Centralize execution in a pipeline with OIDC (episode 13).
• Test the recovery procedure. A backup never tested for restore is just storage. Once a quarter, pull state in the staging environment, push it back, and make sure plan is clean.
• Keep state free of unnecessary secrets. State will always store secrets (episode 15) — but minimize the amount of secrets, use references (e.g. ARNs to Vault) when possible, and protect the bucket with strict IAM.

#Common Mistakes in DR & State Recovery

#Conclusion

In this episode 19 you've built the last line of defense for your IaC code: disaster recovery for state. We discussed why state backup is the DR foundation, how S3 versioning + DynamoDB locking form a layered defense, the recovery procedure when state corrupts (diagnose → pull → repair → push → verify), and the last-resort rescue path of re-importing with import blocks when state is truly lost. You also got a DR runbook checklist and best practices that must be upheld as team discipline.

Key takeaways to bring home:

• State is Terraform's memory — protecting it means protecting control over the infrastructure.
• S3 versioning is instant rollback; the DynamoDB lock prevents simultaneous corruption.
• state pull for inspection & backup, state push for recovery — with backup and validation in both directions.
• When state is totally lost, re-import with import blocks (episode 8) is always the way out.

You now have the entire tactical toolkit: from writing code, managing state, to rescuing it from a fire. Now it's time to assemble everything into one big picture — how all the pieces you learned from episode 0 to 19 come together in a complete enterprise IaC architecture.

In episode 20 — the closing episode of the Learn Terraform series — we'll design an end-to-end enterprise architecture: modular + Terragrunt, a centralized remote backend, passwordless OIDC authentication, secrets from Vault, an OPA policy gate, and automatic drift detection — plus a production readiness checklist and a code review guide. This is the peak of your entire journey. Stay excited!