Learn Terraform - Infrastructure Drift Detection & Refactoring

#Introduction

After building the complete multi-tier infrastructure stack in episode 17 — VPC, EKS, RDS, and encrypted S3 — in this episode we'll face one of the biggest enemies of all the IaC code you've written: infrastructure drift.

This is a very common story in the real world. One Monday morning, the monitoring team reports cloud costs mysteriously rising. After an investigation, it turns out an engineer — with good intentions — opened the AWS Console, went to the EC2 page, and resized the production instance from t3.micro to t3.medium because "the application seems a bit slow". They didn't get around to changing the Terraform code, didn't open a Pull Request, and didn't tell anyone. The next day, when another team runs terraform plan, Terraform shows changes they never planned — and now the configuration (code), state, and real cloud condition are all in conflict.

This phenomenon is called infrastructure drift: the gap between what's declared in the code, what's recorded in state, and what actually exists in the cloud. It happens not because your code is wrong, but because the real world keeps changing — and humans, with all their ClickOps, are the most unpredictable source of change.

Why does this matter? Because the core promise of IaC is reproducibility: infrastructure must be able to be rebuilt exactly from code. As long as there's drift, that promise is broken — and a previously-safe terraform apply could suddenly destroy intentional changes, or let inconsistencies pile up until nobody knows the true state. In this episode you'll learn to detect drift automatically, decide when to reconcile or adopt changes, and refactor code without downtime. This is the skill that separates Terraform users from engineers who operate Terraform.

#Main Discussion

#Understanding Infrastructure Drift

In a healthy Terraform system, there are three sources of truth that must be aligned:

Drift occurs when the real condition deviates from what's recorded in state (and ultimately from the code). Before plan or apply, Terraform always does a refresh — re-reading the object's condition from the cloud API and comparing it with state. If there's a difference, that's drift. If there's a difference between state and code, that's a planned change.

Imagine three people reading one cooking recipe. The code is the written recipe, state is the cook's notes about what they've cooked, and the cloud is the dish actually served. If a guest secretly adds salt to the pot (ClickOps), the cook won't know until they taste it again — and that's exactly what terraform plan is, that "taste test".

There are several common sources of drift:

• ClickOps: manual changes in the cloud console — resizing instances, changing security groups, adding tags.
• Autoscaling / cloud-native resizing: resources changed by another service (e.g. ASG replacing instances, automatic RDS scaling).
• Other people (other teams): changes through tooling outside Terraform — CloudFormation, direct CLI, or Terraform from a different directory.
• Provider drift: on some clouds, the provider normalizes values (e.g. changing tag casing), so state keeps "not matching" even though nothing changed.

#A Drift Scenario: Someone Changes a Resource in the Console

Let's look at drift in its most concrete form. Take an EC2 instance declared in code:

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
 
  tags = {
    Name = "web-server"
  }
}

One day an engineer logs into the AWS Console, opens this instance, and changes its instance type to t3.medium — say, to test whether the application gets faster. They don't change any code. Now there's drift: the code says t3.micro, the cloud says t3.medium.

When you run terraform plan, Terraform refreshes state from the cloud then compares it with the configuration. The result:

aws_instance.web: Refreshing state... [id=i-0abcd1234efgh5678]
 
Terraform will perform the following actions:
 
  # aws_instance.web will be updated in-place
  ~ resource "aws_instance" "web" {
      ~ instance_type      = "t3.medium" -> "t3.micro" # forces replacement
        tags               = {
            "Name" = "web-server"
        }
      ~ vpc_security_group_ids = [
          + "sg-0abc12345def67890",
        ]
    }
 
Plan: 0 to add, 1 to change, 0 to destroy.

Note the ~ sign in front of instance_type: that means Terraform found a difference between the real cloud condition (t3.medium) and the code's desire (t3.micro). The + on the security group means there's a security group manually added in the console that isn't in the code. This is drift — and plan successfully "caught them red-handed".

#Detecting Drift Manually: plan, refresh, and plan -refresh-only

The most basic way to detect drift is running terraform plan periodically. But there's an important nuance: in the normal flow, plan doesn't just detect — it also computes what must change to close the drift. There are three variants you should master:

terraform refresh is an inheritance from old Terraform that is now not recommended for routine use. The problem: refresh rewrites state to match the cloud — meaning it actually hides drift by treating the cloud condition as the truth. That's useful in specific cases (e.g. adopting an intentional change), but if run carelessly, you lose the chance to see the difference. Instead, use terraform plan -refresh-only, which shows state changes explicitly and is safe to review:

terraform plan -refresh-only

The output will show the changes that occur in state (e.g. instance_type changing in state), with no intent to change resources in the cloud.

#terraform plan -detailed-exitcode: A Machine Language for Drift

The problem with a regular terraform plan: it returns exit code 0 on success — whether there are changes or not. Humans can read the output, but machines (CI/CD pipelines, schedulers) can't tell the difference. That's where -detailed-exitcode comes in.

terraform plan -detailed-exitcode
echo "Exit code: $?"

With this flag, Terraform returns one of three exit codes:

#Automating Drift Detection with a Scheduled Pipeline

Manual drift detection means relying on someone's discipline to remember running terraform plan every day — and human discipline is the easiest guard to leak. The right solution is a scheduled pipeline: a CI/CD job that runs automatically every night, runs terraform plan -detailed-exitcode, and reports the drift status.

Here's an example GitHub Actions workflow that runs drift detection every day at 2:00 AM for the production environment:

name: Drift Detection
 
on:
  schedule:
    - cron: '0 2 * * *'          # Every day at 02:00 UTC
  workflow_dispatch:              # Can be triggered manually
 
permissions:
  id-token: write
  contents: read
  issues: write
 
jobs:
  detect-drift:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra/environments/production
    steps:
      - uses: actions/checkout@v4
 
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.9.0"
 
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gh-oidc-terraform-prod
          aws-region: ap-southeast-1
 
      - name: Terraform Init
        run: terraform init
 
      - name: Terraform Plan (detect drift)
        id: plan
        run: |
          set +e
          terraform plan -detailed-exitcode -out=tfplan
          code=$?
          set -e
          echo "terraform_exit_code=$code" >> "$GITHUB_OUTPUT"
          if [ "$code" -eq 2 ]; then
            echo "drift_found=true" >> "$GITHUB_OUTPUT"
          else
            echo "drift_found=false" >> "$GITHUB_OUTPUT"
          fi
 
      - name: Report drift to an issue
        if: steps.plan.outputs.drift_found == 'true'
        uses: actions/github-script@v7
        with:
          script: |
            const body = `Drift detected in \`production\` at \`${new Date().toISOString()}\`.
 
            Run \`terraform plan\` in the directory \`infra/environments/production\`
            to see the details of the changes, then decide: **reconcile** (apply)
            or **adopt** (change the code). Don't let drift accumulate!`;
            github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: "🚨 Drift detected: production",
              body: body
            })

A few details to understand:

• cron: '0 2 * * *' — schedules the job in the early hours, when team change traffic is low so drift notifications don't get drowned out.
• set +e / set -e — temporarily disables exit-on-error so we can capture exit code 2 without failing the job immediately. Without this, a terraform plan returning 2 would stop the workflow instantly.
• drift_found as an output — the drift status is exposed as a variable so the next step can react: create an issue, send a Slack notification, or flag a PR.
• OIDC (role-to-assume) — the job runs passwordless with ephemeral credentials (episode 13), so there's no Access Key stored in workflow secrets.

#Drift Remediation: Reconcile or Adopt

When drift is detected, you face two options, each going in an opposite direction:

Reconcile is done when the console change was a mistake or unwanted — for example an instance resize that turns out to inflate costs. Just run terraform apply, and Terraform returns the resource to the shape declared in the code:

terraform plan -detailed-exitcode -out=tfplan   # make sure the detected drift is indeed what's intended
terraform apply tfplan

Adopt is done when the console change turns out to be the right decision (e.g. a new security group that was genuinely needed, or a new instance type that genuinely fits better). Then you write that change back into the code — so the code becomes the single source of truth again, and apply no longer tries to close the drift:

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.medium"

This adoption must go through the normal flow: change the code → open a Pull Request → reviewed → merged → apply pipeline (episode 13). That way, the change stays recorded in Git and can be audited — not a secret lost inside the console.

#Refactoring Code Without Downtime

Besides external drift, there's another type of "change" that's just as important: code refactoring — tidying up structure, renaming, or moving resources into modules. Naively, if we only rename a resource in the code, Terraform will think the old resource is gone and create a new one — destroy and recreate, which triggers downtime and data loss.

Fortunately, all the mechanisms you need were already learned in episodes 7 and 8. Let's summarize the three main weapons for refactoring without downtime:

#1. moved Blocks: Refactoring Written in Code

When you rename a resource or move it into a module, the moved block tells Terraform: "the object at the old address now lives at the new address — treat both as the same thing."

moved {
  from = aws_instance.web
  to   = aws_instance.nginx
}

During terraform plan, the output will show Moved: aws_instance.web → aws_instance.nginx with Plan: 0 to add, 0 to change, 0 to destroy — no destruction, only relocation within state. After a successful apply, the moved block can be removed from the code.

#2. terraform state mv: Direct Refactoring in the Terminal

For large restructures — for example moving dozens of resources into a module at once — terraform state mv works directly in the terminal without needing to change the code first:

terraform state mv 'module.web.aws_instance.app' 'aws_instance.legacy'

Remember: state mv only moves within state — the configuration code must also be updated to the new address afterward, otherwise the next plan will think the old resource is gone.

#3. lifecycle.create_before_destroy: Replacing Resources Without a Service Gap

There are times when refactoring requires resource replacement (e.g. changing an incompatible AMI, or changing a forces replacement attribute). To ensure there's no service gap, set up the lifecycle so the new resource is created before the old one is destroyed:

resource "aws_instance" "web" {
  ami           = "ami-0updated123456789"
  instance_type = "t3.medium"
 
  lifecycle {
    create_before_destroy = true
  }
}

With create_before_destroy = true, Terraform creates the new instance first, waits until it's ready, then destroys the old one — following the order guaranteed by the dependency graph. This is the core mechanism behind blue-green and rolling replacement strategies at the resource level.

moved {
  from = aws_instance.web
  to   = aws_instance.nginx
}

terraform state mv aws_instance.web aws_instance.nginx

#Common Mistakes in Handling Drift & Refactoring

#Conclusion

In this episode 18 you've understood infrastructure drift — the gap between code, state, and the real cloud condition caused by manual console changes — and three ways to manage it: detecting with terraform plan (and plan -refresh-only), automating detection with terraform plan -detailed-exitcode in a scheduled pipeline, and deciding between reconcile and adopt remediation. You also revisited and strengthened the refactoring without downtime techniques using moved blocks, terraform state mv, and lifecycle.create_before_destroy.

Key takeaways to bring home:

• Drift is inevitable — what sets professional teams apart is automatic detection, not the absence of drift.
• The 0/1/2 exit codes from plan -detailed-exitcode are a machine language schedulers can use to raise alarms.
• Drift detection must stop at reporting to a human; remediation still goes through the review flow.
• Reconcile or adopt? Change the cloud, or change the code — never blur the two.

Drift is a sign that "life goes on" in your infrastructure. But there's one threat far more frightening than drift: a state file that's corrupted or completely lost. When that happens, Terraform loses its memory — and the entire infrastructure becomes one big question: "does this resource still exist?"

In the next episode 19 we'll discuss Disaster Recovery (DR) & State Recovery — recovery procedures when state is corrupted or lost, backup & versioning strategies on remote storage, and the terraform state pull and terraform state push operations for safe manual recovery. Stay excited!