Learn Terraform - Secret Management & State Security

#Introduction

After discussing Managed IaC Platforms like Terraform Cloud and Spacelift in episode 14, which provide remote runners, state management, and policy enforcement — in this episode we'll touch on the topic that most often keeps DevOps teams up at night: secret management and state file security.

Imagine this scenario: a new engineer on your team follows a tutorial on the internet and runs terraform apply on their laptop. A few months later, another team discovers that the terraform.tfstate file got committed to the repository because the .gitignore didn't exist yet. In that JSON file, the production RDS database password, IAM credentials, and private keys are printed out in raw form. The repository might have been private at the time, but privilege escalation from read access, a departing contractor, or a repo fork could open up the entire production environment in minutes.

This story isn't fiction. In episode 5 we already touched on how state files store sensitive data in plaintext — now we'll seriously discuss why that happens, how to secure the storage, and how secrets should be managed inside Terraform so they're never written in code or state.

#Main Discussion

#Why Does the State File Become a "Secret Vault" in Plaintext?

The source of the problem is simple: the state file stores every attribute of the resources Terraform manages, including secret attributes. When an aws_db_instance resource is created with a password argument, that password value is copied as-is into terraform.tfstate in plain JSON format without encryption.

{
  "version": 4,
  "terraform_version": "1.9.5",
  "resources": [
    {
      "mode": "managed",
      "type": "aws_db_instance",
      "name": "app",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "id": "db-XXXXXXXXXXXXX",
            "address": "app.ckzxxxxxxxx.ap-southeast-1.rds.amazonaws.com",
            "username": "appadmin",
            "password": "S3ns1t1f-P4ssw0rd-RDS#2026",
            "port": 5432,
            "engine": "postgres"
          }
        }
      ]
    }
  ]
}

Note the "password": "S3ns1t1f-P4ssw0rd-RDS#2026" line — that's an example of leaking data. It's not just RDS: IAM credentials (aws_iam_access_key), private keys (aws_key_pair), provider tokens — all of it can be in here.

There's one more thing that's often forgotten: the plan file. When you run terraform plan -out=tfplan, the plan result also stores sensitive values in a binary format that anyone with read access can decrypt using terraform show. tfplan must be treated just as carefully as state.

#Securing State: Encryption At-Rest, KMS, and Strict IAM

Since state is the most valuable asset, its security strategy is layered. On AWS, the layers are:

The following backend configuration uses a dedicated KMS key for state encryption:

terraform {
  backend "s3" {
    bucket         = "myapp-tfstate-bucket"
    key            = "production/rds/terraform.tfstate"
    region         = "ap-southeast-1"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:ap-southeast-1:123456789012:key/mykmskey"
    dynamodb_table = "myapp-tfstate-lock"
  }
}

Note the kms_key_id attribute: all state stored in this bucket is encrypted by the team's own KMS key — not AWS's default key. With enable_key_rotation = true on the KMS key, the layered encryption rotates automatically.

Here's the complete state bucket definition along with its policy. The state bucket is usually bootstrapped by hand (or via one separate Terraform directory) because it's the foundation of everything:

resource "aws_s3_bucket" "terraform_state" {
  bucket        = "myapp-tfstate-bucket"
  force_destroy = false
 
  tags = {
    Name        = "Terraform State"
    Environment = "Security"
  }
}
 
resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
 
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.terraform_state.arn
      sse_algorithm     = "aws:kms"
    }
    bucket_key_enabled = true
  }
}
 
resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
 
  versioning_configuration {
    status = "Enabled"
  }
}
 
resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
 
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
 
resource "aws_kms_key" "terraform_state" {
  description             = "KMS key for state file encryption"
  enable_key_rotation     = true
  deletion_window_in_days = 30
}

#Restricting Access with Bucket Policy + IAM

Encryption alone isn't enough — we also have to limit who can read and write state. The principle is least privilege: only the roles running Terraform (CI/CD or senior engineers) may touch this bucket. Add a bucket policy that forces all access through HTTPS:

data "aws_iam_policy_document" "tfstate_policy" {
  statement {
    sid    = "AllowTerraformRunners"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = var.tf_runner_roles # ARN of CI/CD role + approved engineers
    }
    actions = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject",
      "s3:ListBucket",
    ]
    resources = [
      aws_s3_bucket.terraform_state.arn,
      "${aws_s3_bucket.terraform_state.arn}/*",
    ]
  }
 
  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    principals { type = "*" }
    actions   = ["s3:*"]
    resources = [
      aws_s3_bucket.terraform_state.arn,
      "${aws_s3_bucket.terraform_state.arn}/*",
    ]
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}
 
resource "aws_s3_bucket_policy" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  policy = data.aws_iam_policy_document.tfstate_policy.json
}

For GCP, the principle is the same: use the backend "gcs" with a bucket that enables Object Versioning, and restrict access via a dedicated IAM role. Encryption in GCS is enabled automatically (CSEK/CMEK if needed).

#Never Hardcode: Integration with a Secret Manager

The second layer of secret security is the secret source itself. Many teams write passwords directly in variables.tf with default = "rahasia123" or in terraform.tfvars which turns out to get committed. The rules you must hold onto:

1. Never store secrets in code (default) or in committed files.
2. .gitignore .tfvars and state files from day one.
3. Store secrets in a dedicated secret store (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or encrypted SOPS files), then pull them at runtime.

# State — never goes into git
*.tfstate
*.tfstate.backup
*.tfstate.lock.info
 
# Per-environment secret values
*.tfvars
*.tfvars.json
.terraform/
 
# But keep example templates committed
!example.tfvars

Now we'll look at three popular ways to pull secrets from outside the code: HashiCorp Vault, AWS Secrets Manager, and SOPS.

#1. HashiCorp Vault (Data Source)

Vault is the most flexible secret store in the multi-cloud ecosystem. Terraform pulls secrets through a data source, so the values never exist in the HCL code:

provider "vault" {
  address = "https://vault.internal.example.com:8200"
  token   = var.vault_token
}
 
data "vault_kv_secret_v2" "database" {
  mount = "secret"
  name  = "production/database"
}
 
resource "aws_db_instance" "app" {
  engine         = "postgres"
  instance_class = "db.t4g.medium"
 
  username = data.vault_kv_secret_v2.database.data["username"]
  password = data.vault_kv_secret_v2.database.data["password"]
 
  vpc_security_group_ids = [var.app_security_group_id]
  db_subnet_group_name   = var.db_subnet_group_name
 
  skip_final_snapshot = true
}

Vault's strengths: it supports dynamic secrets (e.g. database credentials that rotate automatically), leases, and modern auth methods like Kubernetes ServiceAccount or AWS IAM. The username/password values still end up in state, but the source never leaks in Git.

#2. AWS Secrets Manager (Data Source)

For teams that are entirely on AWS, Secrets Manager is the easiest choice because it's natively integrated and supports AWS-managed automatic rotation:

data "aws_secretsmanager_secret" "database" {
  name = "production/database"
}
 
data "aws_secretsmanager_secret_version" "database" {
  secret_id = data.aws_secretsmanager_secret.database.id
}
 
locals {
  db_secrets = jsondecode(data.aws_secretsmanager_secret_version.database.secret_string)
}
 
resource "aws_db_instance" "app" {
  engine         = "postgres"
  instance_class = "db.t4g.medium"
 
  username = local.db_secrets["username"]
  password = local.db_secrets["password"]
 
  vpc_security_group_ids = [var.app_security_group_id]
  db_subnet_group_name   = var.db_subnet_group_name
 
  skip_final_snapshot = true
}

Since Secrets Manager stores secrets as JSON strings, we use jsondecode(...) then pull the fields via local.db_secrets. Access to the secret is controlled by IAM policy — so make sure the role running Terraform is actually allowed to read the required secret.

#3. SOPS (Encrypted File in Git)

Sometimes teams don't want to depend on additional secret manager infrastructure. SOPS (Secrets OPerationS) from Mozilla lets us store encrypted .tfvars files in Git, with keys managed via age/PGP/KMS. Encryption is done per-field, so Git only sees ciphertext.

creation_rules:
  - path_regex: \.tfvars$
    age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Edit the encrypted file (editor opens, auto-encrypts on save)
sops terraform.tfvars
 
# Or encrypt an existing plaintext file
sops --encrypt --in-place terraform.tfvars
 
# Decrypt without writing to disk, then pipe to Terraform
sops --decrypt terraform.tfvars | terraform plan -var-file=-
sops --decrypt terraform.tfvars | terraform apply -var-file=-

#Comparing Secret Sources

Let's compare the three approaches above in one view:

provider "vault" {
  address = "https://vault.internal.example.com:8200"
}
 
data "vault_kv_secret_v2" "db" {
  mount = "secret"
  name  = "production/database"
}
 
resource "aws_db_instance" "app" {
  username = data.vault_kv_secret_v2.db.data["username"]
  password = data.vault_kv_secret_v2.db.data["password"]
}

data "aws_secretsmanager_secret_version" "db" {
  secret_id = "production/database"
}
 
locals {
  db = jsondecode(data.aws_secretsmanager_secret_version.db.secret_string)
}
 
resource "aws_db_instance" "app" {
  username = local.db["username"]
  password = local.db["password"]
}

# Encrypt tfvars so it's safe to commit to Git
sops --encrypt --in-place terraform.tfvars
 
# Decrypt & pipe directly to Terraform
sops --decrypt terraform.tfvars | \
  terraform apply -var-file=-

#Marking Secrets with sensitive = true

Even though sensitive = true isn't protection for state, it's still mandatory to use. Its function: prevent secret values from appearing in terminal output, CI/CD logs, and plan artifacts. Let's look at the practice.

First, mark the variables that contain secrets:

variable "db_username" {
  type        = string
  description = "Database connection username."
  sensitive   = true
}
 
variable "db_password" {
  type        = string
  description = "Database connection password."
  sensitive   = true
}
 
variable "db_host" {
  type        = string
  description = "Database host address (not a secret)."
  default     = "localhost"
}

Second, mark sensitive outputs:

output "db_endpoint" {
  description = "Production database endpoint."
  value       = aws_db_instance.app.endpoint
  sensitive   = true
}

When terraform apply is run, this value is hidden from the terminal:

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
 
Outputs:
 
db_endpoint = <sensitive>

Even though it's not visible, the value is still stored in state and can be read with terraform output -json by those who have access. So remember: sensitive = true is a visual safety screen, not encryption.

#Common Mistakes in Secret Management & State Security

#Conclusion

In this episode 15 we learned that the state file is a plaintext secret vault — it stores passwords, credentials, and private keys as-is. We also saw how to secure it in layers: at-rest encryption with KMS, versioning, public access blocks, and strict IAM applying least privilege — because anyone who can read state can read all production secrets. Finally, we discussed secret integration from HashiCorp Vault, AWS Secrets Manager, and SOPS, as well as marking variables and outputs with sensitive = true so secret values don't appear in the terminal and logs.

Key takeaways to bring home:

• sensitive = true only masks the display, it is not encryption — state still stores the original value.
• At-rest encryption + versioning + strict IAM are the foundation of state security.
• Never hardcode secrets; pull them from a secret store or an encrypted file.
• .gitignore for *.tfvars and *.tfstate* must be created from day one.
• Plan files (tfplan) are just as sensitive as state.

Now secrets are stored neatly and state is secure. But what if someone else on the team opens an SSH port to the public, or creates a public bucket in the production environment? Manual approval alone won't be enough.

In the next episode 16 we'll discuss Policy as Code (PaC) with OPA & Sentinel — an automatic way to stop terraform apply if the code violates security rules or organizational cost standards. Stay excited!