Learn OpenTofu - Infrastructure Drift Detection & Auto-Remediation

#Introduction

In the previous episode 17 we designed a complete multi-cloud infrastructure — AWS, GCP, Cloudflare, down to Kubernetes — all declared as code. But a good declaration has a silent enemy: drift. Cloud reality can diverge from the code without us realizing it, and if left alone, drift erodes our trust in IaC.

In this episode 18, we'll discuss Infrastructure Drift Detection & Auto-Remediation. We'll learn to detect mismatches between cloud reality and the state file, automate periodic checks with tofu plan -detailed-exitcode, and design safe remediation strategies so infrastructure always returns to its declared state.

#What Is Infrastructure Drift?

Infrastructure drift is the condition where a resource's configuration in the cloud has already diverged from what's recorded in the state file and declared in the code. The most common source of drift is manual ClickOps: an engineer enters the cloud console, changes a security group, adds an instance, or disables encryption — all without changing the code.

Think of the state file as a recipe and the cloud as the finished dish. Drift is when the kitchen silently swaps ingredients — salt for sugar — without telling us. The dish looks the same but tastes different, and we only notice when customers complain.

#Drift Impacts That Are Often Forgotten

Drift isn't just an aesthetic problem. Some of its impacts:

• Loss of control — resources not recorded in state become "orphaned," with nobody managing their lifecycle.
• Security risk — deviating configurations are often looser than the standard.
• Unexpected plans — the next tofu plan shows unexpected changes that confuse reviewers.
• Hard rollbacks — when an incident happens, state no longer reflects reality, making recovery a puzzle.

For that reason, drift detection must run continuously, not just when an engineer remembers to run tofu plan.

#Detecting Drift with -detailed-exitcode

The ordinary tofu plan command always returns exit code 0 when a plan is successfully produced — regardless of whether there are changes. Yet we need to distinguish three states. That's where the -detailed-exitcode flag comes in:

tofu plan -detailed-exitcode -out=plan.tfplan
echo "Exit code: $?"

Exit code: 2

The -detailed-exitcode flag makes tofu plan far more informative:

Exit code 2 is the main drift signal. By checking this code programmatically, machines can know when infrastructure deviates without reading long text output.

#Automating Drift Detection with a Scheduled Pipeline

Running tofu plan manually isn't a strategy. We automate it with GitHub Actions using a schedule trigger with cron — building on the framework we created in episode 13:

name: drift-detect
 
on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch:
 
permissions:
  id-token: write
  contents: read
 
jobs:
  detect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - uses: opentofu/setup-opentofu@v1
 
      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-oidc
          aws-region: ap-southeast-1
 
      - name: Detect drift
        id: plan
        run: |
          tofu init
          tofu plan -detailed-exitcode -no-color > plan.log
          status=$?
          echo "plan_exit=$status" >> "$GITHUB_OUTPUT"
          if [ "$status" -eq 2 ]; then
            echo "drift-found=true" >> "$GITHUB_OUTPUT"
          fi

This workflow runs every morning at 6 AM and can also be triggered manually. The interesting part: with the schedule expression, the pipeline detects drift without waiting for a PR. If status is 2, the drift-found variable becomes true and is ready for the next step to consume.

#Automatic Remediation Strategies

Finding drift is half the work; fixing it is the other half. There are several strategies with different levels of aggressiveness:

#Strategy 1: Sync State Only

If drift comes from small, intentionally desired changes in the cloud, we can record them in state without changing resources:

tofu apply -refresh-only

The tofu apply -refresh-only command updates state to match cloud reality without changing a single resource. Useful when cloud changes were made legitimately and we only want state to follow.

#Strategy 2: Auto-Apply Plan

If we're confident the code is the source of truth, drift is remediated by applying the detected plan:

      - name: Auto-remediate drift
        if: steps.plan.outputs.drift-found == 'true'
        run: |
          tofu apply plan.tfplan -auto-approve

This flow automatically returns infrastructure to its declared state. This is the true meaning of convergence: a system continuously moving toward an ideal state.

#Strategy 3: Notification and Review

For critical infrastructure, full automatic remediation may be too risky. The alternative is sending notifications to the team — for example, adding a step that opens an issue in the repository or sends a message to Slack — then waiting for a human to decide. This strategy keeps humans in the loop, not out of it.

#Drift Detection Best Practices

A few things to keep the drift detection system healthy:

• Run detection in every environment, not just production.
• Save the plan log as a pipeline artifact for auditing.
• Separate drift from planned changes — check whether drift-found appears outside the deploy window.
• Combine with prevent_destroy and lifecycle for resources that must never be deleted automatically.
• Use remote state data sources so each environment checks its own state.

#Conclusion

In episode 18, we discussed Infrastructure Drift Detection & Auto-Remediation. We learned what drift is, how tofu plan -detailed-exitcode distinguishes three states, how to automate checks with a scheduled pipeline, and remediation strategies from passive to aggressive.

Key takeaways:

• Drift is a mismatch between cloud reality, state, and code — usually caused by ClickOps.
• tofu plan -detailed-exitcode returns exit code 0, 1, or 2.
• Exit code 2 means there are changes and could be a sign of drift.
• A scheduled pipeline with schedule makes detection run automatically every day.
• tofu apply -refresh-only syncs state without changing resources.
• Auto-apply returns infrastructure to the ideal state, but must be limited to non-critical environments.

Automated detection and remediation keep your infrastructure always under control, even when humans make mistakes in the console. But all these systems mean nothing if your state is corrupted or lost. In the next episode, episode 19, we'll cover Disaster Recovery & Encrypted State Recovery — how to recover when the worst happens. See you there!