Learn Terraform - Provisioning Complex Cloud Infrastructure Stack

#Introduction

After discussing Policy as Code with OPA and Sentinel in episode 16 — installing an automatic guard rail before terraform apply — in this episode we'll assemble everything learned from episode 0 to 16 into one complete whole: designing and provisioning a complete multi-tier cloud infrastructure stack from scratch.

So far you've gotten to know resources, variables, state, modules, CI/CD, and policy gates separately. But in the real working world, infrastructure never stands alone: applications need a network, the network needs compute, compute needs a database, and all of it needs object storage for artifacts. A product team usually gets a request like this:

"Please prepare a production environment: multi-AZ VPC, Kubernetes cluster, PostgreSQL database, and a bucket for build artifacts — all private, encrypted, and following best practices."

This is the episode where you'll learn to answer that request. We'll build a multi-tier stack on AWS using community modules and a local module, understand why each layer is structured that way, look at the plan summary, and discuss cost considerations, security group security, and best practices for production scale.

#Main Discussion

#The Multi-Tier Architecture We'll Build

Our architecture consists of four layers (tiers), each with a different responsibility. All components sit inside one private VPC, except for the load balancer and NAT which genuinely need controlled public access.

The data flow: incoming traffic goes through the Internet Gateway → public load balancer → services inside the EKS cluster → the application reads/writes to RDS (in the private subnet) and S3. Outbound communication (e.g. pulling images from a registry) exits through the NAT Gateway. No compute or database resource has a public IP.

#Code Organization & Directory Structure

We'll use the directory-based multi-environment pattern from episode 10 — one directory per environment. The complete structure:

infrastructure/
├── environments/
│   └── production/
│       ├── main.tf            # Root module: orchestration of all layers
│       ├── backend.tf         # S3 + DynamoDB (from episode 5)
│       ├── providers.tf       # AWS provider + default_tags
│       ├── variables.tf
│       ├── terraform.tfvars   # Environment values (gitignored, see ep. 15)
│       ├── outputs.tf
│       └── modules/
│           └── rds/           # Local module for PostgreSQL
│               ├── main.tf
│               ├── variables.tf
│               └── outputs.tf

We'll take the VPC and EKS from community modules on the Terraform Registry (terraform-aws-modules/vpc/aws and terraform-aws-modules/eks/aws) because both have been battle-tested thousands of times in production. We'll wrap RDS as a local module to show how to create your own reusable component. We'll write S3 directly in the root since it's simple enough.

#Layer 1: Custom Multi-AZ VPC

The VPC is the foundation of everything. We need three Availability Zones (AZs) for high availability, each with one public and one private subnet. Public subnets host the Internet Gateway route, NAT Gateway, and load balancer; private subnets host all internal workloads.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
 
  name = "${var.project}-vpc"
  cidr = "10.0.0.0/16"
 
  azs             = local.azs
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
 
  enable_nat_gateway = true
  single_nat_gateway = false   # HA: one NAT per AZ
  enable_vpn_gateway = false
 
  enable_dns_hostnames = true
  enable_dns_support   = true
 
  map_public_ip_on_launch = true
 
  tags = {
    Tier = "Network"
  }
}

Several important decisions here and their reasons:

• cidr = "10.0.0.0/16" — a large enough private address space, with /24 subnets per AZ. Never use a CIDR that conflicts with an on-premise network or another VPC peering.
• single_nat_gateway = false — with a single NAT gateway, all AZs share one egress path; if that gateway goes down, the private AZs depending on it lose internet access. For production, install one NAT per AZ (NAT per AZ, egress per AZ).
• Public subnets — the home of the Internet Gateway and public load balancer; private subnets — the home of EKS, RDS, and all internal workloads.

#Layer 2: Managed Kubernetes Cluster (EKS)

The compute layer uses Amazon EKS — managed Kubernetes. The EKS community module prepares the control plane, node groups, IAM roles, and security groups automatically. Our cluster is deliberately private (public endpoint disabled) because all access happens from inside the VPC.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"
 
  cluster_name    = "${var.project}-eks"
  cluster_version = "1.31"
 
  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets
 
  cluster_endpoint_public_access  = false
  cluster_endpoint_private_access = true
 
  eks_managed_node_groups = {
    general = {
      desired_size   = 3
      min_size       = 3
      max_size       = 10
      instance_types = ["t3.large"]
      capacity_type  = "ON_DEMAND"
      subnet_ids     = module.vpc.private_subnets
    }
 
    spot = {
      desired_size   = 2
      min_size       = 1
      max_size       = 8
      instance_types = ["m5.large", "m5a.large"]
      capacity_type  = "SPOT"
      subnet_ids     = module.vpc.private_subnets
    }
  }
 
  tags = {
    Tier = "Compute"
  }
}

The node group pattern above is a common production pattern:

• general node group (On-Demand) — hosts stateful/critical workloads that must not be interrupted (e.g. controllers, database tools).
• spot node group — hosts stateless and batch workloads that can tolerate interruption, at up to ~60-90% lower cost.

#Layer 3: Managed PostgreSQL Database (Multi-AZ RDS)

The database is the most "sacred" component — losing data is unforgivable. That's why we wrap RDS as a reusable local module (modules/rds) so other environments (staging, dev) can use it with different inputs.

variable "identifier" {
  type = string
}
 
variable "instance_class" {
  type    = string
  default = "db.r6g.large"
}
 
variable "vpc_id" {
  type = string
}
 
variable "subnet_ids" {
  type = list(string)
}
 
variable "allowed_security_group_ids" {
  type    = list(string)
  default = []
}
 
variable "db_name" {
  type = string
}
 
variable "username" {
  type      = string
  sensitive = true
}
 
variable "password" {
  type      = string
  sensitive = true
}

resource "aws_db_subnet_group" "this" {
  name       = "${var.identifier}-subnet"
  subnet_ids = var.subnet_ids
}
 
resource "aws_security_group" "this" {
  name   = "${var.identifier}-sg"
  vpc_id = var.vpc_id
 
  dynamic "ingress" {
    for_each = var.allowed_security_group_ids
    content {
      from_port       = 5432
      to_port         = 5432
      protocol        = "tcp"
      security_groups = [ingress.value]
    }
  }
 
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
 
resource "aws_db_instance" "this" {
  identifier     = var.identifier
  engine         = "postgres"
  engine_version = "16.3"
  instance_class = var.instance_class
 
  db_name  = var.db_name
  username = var.username
  password = var.password
 
  multi_az                     = true
  db_subnet_group_name         = aws_db_subnet_group.this.name
  vpc_security_group_ids       = [aws_security_group.this.id]
  storage_encrypted            = true
  storage_type                 = "gp3"
  backup_retention_period      = 30
  deletion_protection          = true
  skip_final_snapshot          = false
  final_snapshot_identifier    = "${var.identifier}-final"
  performance_insights_enabled = true
 
  tags = {
    Tier = "Data"
  }
}

output "endpoint" {
  value = aws_db_instance.this.endpoint
}
 
output "security_group_id" {
  value = aws_security_group.this.id
}

Note the dynamic "ingress" block: it creates one ingress rule per allowed security group — so we just mention [module.eks.node_security_group_id] from the root, and only EKS nodes can reach port 5432. This is far safer than opening up a subnet CIDR.

Now call the module from the root:

module "postgres" {
  source = "./modules/rds"
 
  identifier     = "${var.project}-postgres"
  instance_class = "db.r6g.large"
 
  db_name  = var.db_name
  username = var.db_username
  password = var.db_password
 
  vpc_id                    = module.vpc.vpc_id
  subnet_ids                = module.vpc.private_subnets
  allowed_security_group_ids = [module.eks.node_security_group_id]
}

#Layer 4: Object Storage (S3) with KMS & Lifecycle

The last layer is an S3 bucket for build artifacts and backups. We make it totally private, with KMS encryption (automatic key rotation), versioning, and a lifecycle policy to control storage costs over time.

resource "aws_s3_bucket" "artifacts" {
  bucket = "${var.project}-artifacts"
 
  lifecycle {
    prevent_destroy = true
  }
}
 
resource "aws_s3_bucket_versioning" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
 
  versioning_configuration {
    status = "Enabled"
  }
}
 
resource "aws_s3_bucket_server_side_encryption_configuration" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
 
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.artifacts.arn
      sse_algorithm     = "aws:kms"
    }
    bucket_key_enabled = true
  }
}
 
resource "aws_s3_bucket_lifecycle_configuration" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
 
  rule {
    id     = "archive-and-expire"
    status = "Enabled"
 
    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }
 
    transition {
      days          = 90
      storage_class = "GLACIER"
    }
 
    expiration {
      days = 365
    }
  }
}
 
resource "aws_s3_bucket_public_access_block" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
 
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
 
resource "aws_kms_key" "artifacts" {
  description             = "KMS key for the artifacts bucket"
  enable_key_rotation     = true
  deletion_window_in_days = 30
}

The lifecycle logic is simple yet cost-effective: files are stored in STANDARD for the first 30 days, move to STANDARD_IA (infrequent access) after 30 days, to GLACIER after 90 days, and are automatically deleted after 365 days. Since versioning is enabled, every file version follows the same rules.

#Main, Provider, and Outputs

The root file assembles everything, sets the remote backend (from episode 5), and default tags so all resources are automatically labeled:

terraform {
  required_version = ">= 1.6.0"
 
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
 
  backend "s3" {
    bucket         = "myapp-tfstate-bucket"
    key            = "production/fullstack/terraform.tfstate"
    region         = "ap-southeast-1"
    encrypt        = true
    dynamodb_table = "myapp-tfstate-lock"
  }
}
 
provider "aws" {
  region = var.region
 
  default_tags {
    tags = {
      Environment = "production"
      Project     = var.project
      ManagedBy   = "terraform"
    }
  }
}
 
locals {
  azs = ["ap-southeast-1a", "ap-southeast-1b", "ap-southeast-1c"]
}

Outputs for the application or other stacks to consume:

output "vpc_id" {
  value = module.vpc.vpc_id
}
 
output "cluster_name" {
  value = module.eks.cluster_name
}
 
output "cluster_endpoint" {
  value = module.eks.cluster_endpoint
}
 
output "database_endpoint" {
  value     = module.postgres.endpoint
  sensitive = true
}
 
output "artifacts_bucket" {
  value = aws_s3_bucket.artifacts.id
}

#Running & Verifying the Stack

With all files in place, it's time to run the standard workflow:

terraform init             # Download provider + community modules
terraform fmt -recursive   # Format all the code
terraform validate         # Validate syntax & references
terraform plan -out=tfplan # Create the plan
terraform apply tfplan     # Execute

The terraform plan output for a stack like this will be very large — dozens of resources from the VPC and EKS modules. The summary looks roughly like this:

Terraform will perform the following actions:
 
  # module.vpc.aws_vpc.this will be created
  + resource "aws_vpc" "this" {
      + cidr_block           = "10.0.0.0/16"
      + enable_dns_hostnames = true
      + enable_dns_support   = true
      ...
    }
 
  # module.vpc.aws_subnet.private[0] will be created
  + resource "aws_subnet" "private" {
      + availability_zone       = "ap-southeast-1a"
      + cidr_block              = "10.0.1.0/24"
      ...
    }
 
  # module.eks.module.eks_managed_node_group["general"] will be created
  ...
  # module.postgres.aws_db_instance.this will be created
  ...
  # aws_s3_bucket.artifacts will be created
  ...
 
Plan: 78 to add, 0 to change, 0 to destroy.

#Cost, Security, and Best Practices Considerations

This stack runs "forever" and will keep being billed. There are three dimensions you must always consider.

#1. Cost

#2. Security Groups & Network Security

• The database can only be reached from the EKS node security group, not from a subnet CIDR — so application pods are the only clients.
• Private EKS cluster: the public endpoint is disabled; kubectl access goes through the private endpoint or a bastion/jump host.
• Totally private S3 with block_public_access + KMS; there's never a bucket policy opening public access.
• Default tags (Environment, Project, ManagedBy) make audit and automatic cost allocation easy.

#3. Production-Grade Best Practices

#Common Mistakes in Provisioning Complex Stacks

#Conclusion

In this episode 17 we built a complete multi-tier infrastructure stack from scratch on AWS: a multi-AZ VPC with public/private subnets, an Internet Gateway, and per-AZ NAT Gateways; an EKS cluster with on-demand + spot node groups; a multi-AZ PostgreSQL database (RDS) reachable only from EKS nodes; and a KMS-encrypted S3 bucket with versioning and a lifecycle policy. We also laid out an environment-based directory structure, called community and local modules, looked at the plan summary, and discussed cost considerations, security group security, and production-grade best practices.

Key takeaways to bring home:

• A production stack consists of interdependent tiers: Network → Compute → Data → Storage.
• Modules (community & local) keep complex stacks tidy, consistent, and reusable.
• Layered security: private cluster, database only for EKS nodes, totally private storage.
• Deletion protection + encryption + default tags are "mandatory costs" you can't skip.
• Total cost easily breaks thousands of rupiah per month; control it with lifecycle, spot, and separate environments.

You can replicate all the examples above in your own AWS account (use a free-tier-eligible region while practicing) or adapt them to GCP with analogous modules: GKE replaces EKS, Cloud SQL replaces RDS, and GCS with lifecycle management replaces S3.

Now the infrastructure stands strong. But there's one problem lurking quietly: what happens when someone changes infrastructure manually in the cloud console? The state no longer matches reality, and one day terraform plan will show changes you never planned.

In the next episode 18 we'll discuss Infrastructure Drift Detection & Refactoring — detecting differences between the real cloud condition vs state, automating drift detection with a scheduled pipeline, and restructuring code without downtime. Stay excited!