Learn Terraform - Complete Enterprise Production Architecture & Best Practices

#Introduction

After discussing disaster recovery and state recovery in episode 19 — rescuing Terraform's "memory" when the worst happens — in this episode, the final episode of the Learn Terraform series, we'll assemble your entire journey into one complete whole: a production-grade enterprise IaC architecture.

Our journey started in episode 0 with a very simple question: "what is Infrastructure as Code and why does the world need Terraform?" Twenty episodes later, you're no longer asking what — but how to design a system that's alive, secure, and operable at organizational scale. In between, you've learned to write HCL, manage state, build modules, automate pipelines, secure secrets, enforce policies, and rescue state from a fire.

All those skills have felt separate so far. But in the real working world, they run as one system. Imagine a company with dozens of engineers, hundreds of repos, and thousands of cloud resources. Without a unified architecture, each team would make its own "way": one team uses local state, another hardcodes secrets in code, yet another skips review. The resulting chaos isn't merely technical — it becomes business risk.

Episode 20 is the tip of the iceberg. We'll design one end-to-end enterprise architecture that unifies modular architecture + Terragrunt, a remote S3/DynamoDB backend, passwordless OIDC authentication in GitHub Actions, secrets from HashiCorp Vault, an OPA policy gate, and automatic drift detection. Then we'll close with a production readiness checklist, a code review guide, and a full recap of the journey from episode 0. This isn't an episode about features — it's an episode about the way of thinking of a platform engineer.

#Main Discussion

#The Big Picture of a Unified Enterprise Architecture

Before looking at the code, understand the overall flow first. Imagine an engineer wanting to change production infrastructure. The complete flow:

1. The engineer writes code in the monorepo → opens a Pull Request.
2. The PR pipeline (episode 13) runs: fmt → validate → tflint → security scan (Checkov/Trivy, episode 11) → terraform plan → the plan result is posted to the PR.
3. The OPA policy gate (episode 16) evaluates the plan — if there's a violation (public bucket, open port, missing tag), the pipeline is blocked.
4. A human does the code review; approval is the final gate.
5. After merge, the apply pipeline runs — entering the cloud without a password through OIDC (episode 13), pulling secrets from HashiCorp Vault at runtime (episode 15).
6. State is stored centrally on S3 + DynamoDB (episodes 5 & 19), versioned and locked.
7. Every night, drift detection (episode 18) runs plan -detailed-exitcode and reports if the infrastructure has drifted.

All these flows are supported by the modular + Terragrunt structure (episodes 9 & 12) that keeps the code DRY across environments. Let's map each component to its source episode:

#The Enterprise Monorepo Directory Structure

The foundation of the architecture is the repository structure. Here's a monorepo structure commonly used at companies running Terraform at scale:

infra/
├── terragrunt.hcl                 # Global Terragrunt configuration
├── account.hcl                    # Shared variables per account/env
├── environments/
│   ├── dev/
│   │   ├── vpc/terragrunt.hcl
│   │   ├── eks/terragrunt.hcl
│   │   ├── rds/terragrunt.hcl
│   │   └── s3/terragrunt.hcl
│   ├── staging/
│   │   └── ...                    # Same, different values
│   └── production/
│       ├── vpc/terragrunt.hcl
│       ├── eks/terragrunt.hcl
│       ├── rds/terragrunt.hcl
│       └── s3/terragrunt.hcl
├── modules/                       # Internal modules (episode 9)
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── versions.tf
│   ├── eks/
│   ├── rds/
│   └── s3/
├── policies/                      # Policy as Code (episode 16)
│   ├── deny_public_s3.rego
│   ├── require_tags.rego
│   └── deny_public_ssh.rego
├── scripts/
│   ├── opa_gate.sh                # OPA evaluation on tfplan.json
│   └── drift_report.sh
├── .github/workflows/
│   ├── pr-plan.yml                # PR pipeline (episode 13)
│   ├── apply.yml                  # Merge → apply pipeline
│   └── drift-detection.yml        # Scheduled drift check (episode 18)
├── terragrunt-modules/            # (Optional) shared terragrunt units
└── docs/
    ├── runbooks/
    └── inventory.md               # Resource ID list (episode 19)

This structure separates two different things: modules/ contains logic (reusable code), while environments/*/ contains instantiation (values per environment). Terragrunt bridges the two by eliminating the backend and provider configuration duplication that would otherwise be rewritten in every environment directory.

#Centralized Remote Backend (S3 + DynamoDB)

State is stored in one centralized bucket, with one key per component per environment. Terragrunt ensures every unit (vpc, eks, rds, s3) has its own state — not one giant state for everything:

# Generate backend & provider configuration centrally
generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite_terragrunt"
 
  contents = <<EOF
terraform {
  backend "s3" {
    bucket         = "acme-tfstate-global"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "ap-southeast-1"
    encrypt        = true
    dynamodb_table = "acme-tfstate-lock"
  }
}
EOF
}
 
remote_state {
  backend = "s3"
  config = {
    bucket         = "acme-tfstate-global"
    region         = "ap-southeast-1"
    encrypt        = true
    dynamodb_table = "acme-tfstate-lock"
  }
}

include "root" {
  path = find_in_parent_folders()
}
 
dependency "vpc" {
  config_path = "../vpc"
}
 
inputs = {
  engine             = "postgres"
  engine_version     = "16.3"
  instance_class     = "db.r6g.large"
  multi_az           = true
  deletion_protection = true
  vpc_id             = dependency.vpc.outputs.vpc_id
  subnet_ids         = dependency.vpc.outputs.private_subnets
}

Things to note:

• One bucket, many keys — path_relative_to_include() automatically gives each unit a unique key like production/rds/terraform.tfstate. This provides per-component state isolation while keeping configuration consistent.
• dependency — RDS automatically reads the VPC outputs (episode 12). Without Terragrunt, you'd have to write terraform_remote_state data sources repeatedly in every unit.
• One DynamoDB lock table protects all units — the enterprise version of the lesson from episodes 5 and 19.

#Passwordless OIDC Authentication in GitHub Actions

There are no static cloud secrets in the repository. Every job gets ephemeral credentials through OIDC (episode 13) — GitHub issues an ID token, and AWS assumes a specific IAM role for that repo, branch, and environment:

name: Terraform Apply
 
on:
  push:
    branches: [main]
    paths: ["infra/**"]
 
permissions:
  id-token: write          # Key: allow the job to request an OIDC token
  contents: read
 
env:
  TF_IN_AUTOMATION: "1"
  VAULT_ADDR: "https://vault.acme.internal:8200"
 
jobs:
  apply:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra/environments/production
    steps:
      - uses: actions/checkout@v4
 
      - uses: hashicorp/setup-terraform@v3
 
      - name: Configure AWS Credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gh-oidc-terraform-prod
          aws-region: ap-southeast-1
 
      - name: Terraform Init
        run: terraform init
 
      - name: Terraform Apply
        run: terraform apply -auto-approve
        env:
          TF_VAR_vault_token: ${{ secrets.VAULT_TOKEN }}

Note the VAULT_ADDR and TF_VAR_vault_token: cloud credentials come from OIDC, while the Vault token is injected as a workflow secret — again with no static access key written in the code. The gh-oidc-terraform-prod IAM role can only be assumed by jobs from this repo in the production environment — not from a laptop, not from another repo.

#Secret Management: HashiCorp Vault

Secrets are never stored in the code or in state inputs. Vault becomes the single source of secrets, and Terraform pulls them at runtime via a data source (episode 15):

provider "vault" {
  address = var.vault_addr
  token   = var.vault_token
}
 
data "vault_kv_secret_v2" "database" {
  mount = "secret"
  name  = "production/database"
}
 
resource "aws_db_instance" "primary" {
  identifier     = "acme-postgres"
  engine         = "postgres"
  engine_version = "16.3"
  username       = data.vault_kv_secret_v2.database.data["username"]
  password       = data.vault_kv_secret_v2.database.data["password"]
 
  multi_az            = true
  deletion_protection = true
}

The username and password values enter state (unavoidable, episode 15) — but the source never leaks into Git. Plus, Vault supports dynamic secrets and automatic rotation, which static .tfvars secrets can't do.

#The OPA Policy Gate: Block Before Apply

Before apply is allowed, the plan is evaluated against organizational policy. Policies are written in Rego (episode 16) — for example forbidding public buckets, forbidding SSH ports to 0.0.0.0/0, and requiring standard tags:

package terraform
 
import rego.v1
 
deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_security_group"
	rc.change.after.ingress[_].from_port == 22
	rc.change.after.ingress[_].cidr_blocks[_] == "0.0.0.0/0"
	msg := sprintf("%s: SSH port open to the public is forbidden", [rc.address])
}

The gate runs in the PR pipeline — between plan and review:

#!/usr/bin/env bash
set -euo pipefail
 
cd infra/environments/production
 
terraform plan -out=tfplan
terraform show -json tfplan > tfplan.json
 
opa eval \
  --data ../../../policies \
  --input tfplan.json \
  "data.terraform.deny" > result.json
 
if jq -e '.result[].expressions[].value | length > 0' result.json > /dev/null; then
  echo "❌ POLICY BLOCKED — violations found:"
  jq -r '.result[].expressions[].value[]' result.json
  exit 1
fi
 
echo "✅ Policy check passed"

#Automated Drift Detection

The architecture's closing piece: a system that maintains itself. The drift detection workflow from episode 18 is scheduled every night for each environment, using terraform plan -detailed-exitcode:

name: Drift Detection
 
on:
  schedule:
    - cron: '0 3 * * *'
  workflow_dispatch:
 
jobs:
  drift:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra/environments/production
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gh-oidc-terraform-prod
          aws-region: ap-southeast-1
 
      - run: terraform init
 
      - name: Detect drift
        id: plan
        run: |
          set +e
          terraform plan -detailed-exitcode -out=tfplan
          code=$?
          set -e
          if [ "$code" -eq 2 ]; then
            echo "drift_found=true" >> "$GITHUB_OUTPUT"
          else
            echo "drift_found=false" >> "$GITHUB_OUTPUT"
          fi
 
      - name: Open an issue when drift is found
        if: steps.plan.outputs.drift_found == 'true'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: "🚨 Drift detected: production",
              body: "Infrastructure has drifted from the code. Run `terraform plan` and decide: reconcile (apply) or adopt (change the code)."
            })

This is how the unified enterprise architecture closes the loop: code reaches the cloud through layered automatic gates, state is guarded and backed up, secrets never leak, policy can't be bypassed, and drift can't hide for more than one night.

#Production Readiness Checklist

Before your architecture deserves the production-grade label, pass the entire checklist below — adapted from the lessons of the whole series:

#A Code Review Guide for Terraform Code

When reviewing a Terraform Pull Request, don't just check "is the syntax correct?". There's a layer of questions to ask every time:

A practical rule that always applies: a good Terraform PR is small, single-purpose, and comes with the terraform plan result. If a PR changes 40 resources at once, ask for it to be split — a bug in the middle of a big change is far harder to find than a bug in a small, focused change.

#Journey Recap: From Episode 0 to Episode 20

Let's look at the big map you've traversed together over twenty episodes:

From merely understanding the terraform init command in episode 2, you're now able to design a system where an entire company's infrastructure is written as code, verified by machines, secured by policy, and recovered with tested procedures. That's a rare, highly valuable capability in the DevOps, SRE, and Cloud Engineering job market.

#Conclusion

Congratulations — you've completed the Learn Terraform series from episode 0 to episode 20!

Let's briefly recap the meaning of this journey. In this episode 20 we designed a unified enterprise IaC architecture: a modular monorepo structure with Terragrunt, a centralized and versioned S3 + DynamoDB remote backend, passwordless OIDC authentication, secrets from HashiCorp Vault, an OPA policy gate that blocks violations before apply, and drift detection that guards the system every night. We closed it with a production readiness checklist and a code review guide that turn the series' lessons into a single gateway to production.

But the most valuable thing isn't the code — it's the way of thinking you now have:

1. Declarative, not imperative — write desires, not steps. Let the system do the rest.
2. Versioned, not one-off — every change is recorded in Git, reviewable, rollbackable.
3. Secured, not assumed safe — secrets, policies, and access are guarded by layered defenses, not by trust.
4. Operated, not abandoned — drift is detected, state is backed up, recovery is drilled. Infrastructure is a living system.

Your journey doesn't stop here. A few next steps you can take to keep growing:

• Build a real project — take one workload (e.g. a blog or a small API), deploy it with this episode's architecture, and practice the entire production checklist.
• Explore the supporting ecosystem — learn OpenTofu (the open-source Terraform fork), dive deep into Kubernetes with the Learn Kubernetes series, or explore HCP Terraform/Spacelift (episode 14) for managed IaC.
• Deepen operations — combine it with observability (Prometheus/Grafana), GitOps (ArgoCD/Flux), and platform engineering to unify IaC with the application runtime.
• Contribute and share — publish modules to the Terraform Registry, write Rego policies for your organization, or share your experience in writing. Teaching is the best way to truly master something.

Remember the foundation we built since episode 0: a sturdy house stands on a strong foundation. The foundation you've built over these twenty episodes isn't just code — it's trust: trust that infrastructure can be rebuilt, recovered, and accounted for. The engineer who can provide that trust is the engineer organizations rely on.

Thank you for accompanying this journey to the end. Now — open your terminal, write your first module, and make infrastructure something that can be rebuilt with a single command. Happy building, true infrastructure engineer!