Learn Vault - A Complete Production-Grade Vault Architecture Case Study

#Introduction

After building observability and troubleshooting in episode 24 — so our Vault is not only secure and available, but also observable — this episode, the final one of the Learn Secret Management with HashiCorp Vault series, weaves your entire journey into one whole: an end-to-end production-grade enterprise Vault architecture.

This journey began in episode 0 with a simple question: "what is secret management and why does the world need Vault?" Twenty-five episodes later, you're no longer asking what — but how to design a secret management system that is alive, secure, scalable, and operable at organizational scale. In between, you've learned architecture and unsealing, secrets engines (KV, dynamic database, Transit, PKI, TOTP/SSH/AWS), authentication and authorization, identity, leases, response wrapping, integration with applications and Kubernetes, CI/CD and IaC, up to HA, auto-unseal, hardening, multi-tenancy, and observability.

All those skills have felt fragmented so far. In the real working world, they run as one system. This episode designs a single architecture for a fictional company — let's call it PT Nusantara Fintech — that wants to secure all its secrets with Vault: cloud infrastructure, applications on Kubernetes, databases, customer PII data, up to human access. This isn't an episode about features — it's an episode about how a security/platform engineer thinks when designing real secret management.

#Main Discussion

#The Architecture Big Picture: PT Nusantara Fintech

We'll design a system with five interconnected layers:

1. Infrastructure Layer — 3-node Vault HA cluster (Raft + AWS KMS auto-unseal + TLS).
2. Kubernetes Integration — Vault Agent Sidecar Injector + Kubernetes Auth in EKS.
3. Database Security — dynamic PostgreSQL credentials with short TTLs.
4. Application Security — the Transit engine for PII encryption (national ID / credit card numbers).
5. Human Access & Governance — OIDC via Okta, audit logs to Grafana Loki, daily snapshots.

Here's the end-to-end request flow on this architecture:

Mapping each component to its origin episode:

#Layer 1: Infrastructure — HA Cluster + Auto-Unseal + TLS

The architecture's foundation is a 3-node Vault HA cluster with Raft (episode 20), AWS KMS auto-unseal (episode 21), and mandatory TLS (episode 22). Per-node server config:

seal "awskms" {
  region     = "ap-southeast-1"
  kms_key_id = "1234abcd-5678-90ef-ghij-klmnopqrstuv"
}
 
storage "raft" {
  path   = "/opt/vault/data"
  node_id = "vault-node-1"
 
  retry_join {
    leader_api_addr = "https://10.0.0.11:8200"
  }
  retry_join {
    leader_api_addr = "https://10.0.0.12:8200"
  }
  retry_join {
    leader_api_addr = "https://10.0.0.13:8200"
  }
}
 
listener "tcp" {
  address         = "0.0.0.0:8200"
  tls_disable     = false
  tls_cert_file   = "/etc/vault.d/tls/vault.crt"
  tls_key_file    = "/etc/vault.d/tls/vault.key"
  tls_min_version = "tls12"
}
 
api_addr     = "https://10.0.0.11:8200"
cluster_addr = "https://10.0.0.11:8201"
disable_mlock = false
 
telemetry {
  prometheus_retention_time = "24h"
  disable_hostname          = true
}

The architectural decisions at this layer:

• 3 nodes — one node failure is still safe (quorum 2), per the tolerance table from episode 20.
• AWS KMS auto-unseal — no manual unseal; 3-of-5 Recovery Keys stored separately as an emergency plan.
• TLS + disable_mlock = false — encrypted traffic, keys can't be swapped to disk.
• Telemetry active — ready to be scraped by Prometheus (episode 24).

#Layer 2: Kubernetes Integration — Agent Injector + K8s Auth

PT Nusantara Fintech's backend apps run on EKS. Pods hold no secrets at all — they get secrets via the Vault Agent Sidecar Injector (episode 17) automated through annotations. What needs to be prepared first: enabling the Kubernetes auth method in Vault and writing a policy for the application service account.

vault auth enable kubernetes
 
vault write auth/kubernetes/config \
  kubernetes_host="https://EKS_CLUSTER_ENDPOINT:443" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
 
vault write auth/kubernetes/role/backend \
  bound_service_account_names=backend-sa \
  bound_service_account_namespaces=backend \
  policies=backend-k8s \
  ttl=1h

path "database/creds/backend-role" {
  capabilities = ["read"]
}

After that, the Pod deployment just needs annotations added to trigger the injector:

deployment.yaml - injector annotations

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-api
  namespace: backend
spec:
  replicas: 3
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "backend"
        vault.hashicorp.com/agent-inject-secret-dbcreds: "database/creds/backend-role"
        vault.hashicorp.com/agent-inject-template-dbcreds: |
          {{- with secret "database/creds/backend-role" -}}
          DB_HOST={{ .Data.host }}
          DB_USER={{ .Data.username }}
          DB_PASSWORD={{ .Data.password }}
          {{- end -}}
    spec:
      serviceAccountName: backend-sa
      containers:
        - name: backend-api
          image: nusantara/backend-api:1.4.2
          env:
            - name: DB_CONFIG
              value: /vault/secrets/dbcreds

The result: the Pod boots, the injector installs a Vault Agent container as a sidecar, the Agent logs in via K8s auth, fetches dynamic database credentials, and renders them to the file /vault/secrets/dbcreds (episodes 14 & 17). The application reads that file — without touching the Vault API at all, without any token in code.

#Layer 3: Database Security — Dynamic PostgreSQL Credentials

None of PT Nusantara Fintech's database credentials are ever static. Vault opens an administrative connection to PostgreSQL and issues on-demand credentials for each application (episode 5):

vault secrets enable -path=database database
 
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="backend-role,reporting-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.internal:5432/nusantara?sslmode=verify-full" \
  username="vault_admin" \
  password="$(cat /etc/vault.d/db-admin-pass)" \
  verify_connection=true
 
vault write database/roles/backend-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Every time an application (via the sidecar Agent) reads database/creds/backend-role, Vault creates a new PostgreSQL user in real time with a 1-hour TTL, complete with VALID UNTIL ensuring that user dies on its own at the database level even if Vault itself dies. No more permanent passwords rotated manually — rotation is something that automatically happens every hour.

#Layer 4: Application Security — Transit for PII Encryption

Sensitive customer data — national ID numbers, credit card numbers — must not be stored in plaintext in the application database. With the Transit engine (episode 6), PT Nusantara Fintech encrypts PII inside the application while the key is centrally managed by Vault. The data stays in the app; only the key is in Vault.

vault secrets enable -path=transit transit
 
vault write -f transit/keys/pii-key \
  type=aes256-gcm96 \
  auto_rotate_period="720h"

The backend calls Vault for encryption and decryption:

PLAINTEXT_B64=$(printf '%s' "1234-5678-9012-3456" | base64)
 
vault write transit/encrypt/pii-key \
  plaintext="$PLAINTEXT_B64"

Key            Value
---            -----
ciphertext     vault:v1:abc123...xyz

vault write transit/decrypt/pii-key \
  ciphertext="vault:v1:abc123...xyz"

When security demands it, the key rotates without changing the app:

vault write -f transit/keys/pii-key/rotate

Old ciphertext values remain decryptable (versioned keys), and new data is automatically encrypted with the latest version. Rotation becomes a one-command operation, not a data migration. This is the main difference between EaaS and traditional application encryption we covered in episode 6: keys never live in application code.

#Layer 5: Human Access & Governance — OIDC, Audit, and Snapshot

Human access to Vault — engineers, admins, auditors — must not go through static tokens. They log in via OIDC with the company's identity provider (Okta / Azure AD), so access follows employment status: employees leaving the company automatically lose access (episodes 10 & 11).

vault auth enable oidc
 
vault write auth/oidc/config \
  oidc_discovery_url="https://nusantara.okta.com" \
  oidc_client_id="VAULT_CLIENT_ID" \
  oidc_client_secret="REDACTED" \
  default_role="engineer"
 
vault write auth/oidc/role/engineer \
  allowed_redirect_uris="https://vault.nusantara.internal/ui/vault/auth/oidc/oidc/callback" \
  user_claim="email" \
  groups_claim="groups" \
  policies="default"

The identity that logs in — with the default policy plus group-based policies from Okta — forms an Entity with an OIDC alias (episode 11). Okta groups (e.g. platform-team, data-team) are mapped to internal Vault policies, so governance flows from one source: the company's identity management.

All access is recorded by the audit device (episode 22) and shipped to Grafana Loki for querying and alerting:

vault audit enable file file_path=/var/log/vault/audit.log log_raw=false

scrape_configs:
  - job_name: vault-audit
    static_configs:
      - targets:
          - localhost
        labels:
          job: vault-audit
          __path__: /var/log/vault/audit.log

Finally, daily Raft snapshots guarantee recovery during a disaster (episode 22), sent to a separate S3 bucket in another region:

0 2 * * * vault /usr/local/bin/scripts/vault-snapshot-daily.sh
 
# Script contents: vault operator raft snapshot save -> aws s3 cp (separate bucket, KMS-encrypted)

#Vault Production Readiness Checklist

Before your architecture deserves to be called production-grade, pass the entire checklist below — the synthesis of the whole series:

#Journey Recap: From Episode 0 to Episode 25

Let's look at the big map you've traversed together over twenty-six episodes:

From simply running vault server -dev in episode 0, you can now design a system where an entire company's secrets are managed by one platform that is available, measurable, audited, and recoverable. That's a rare skill of very high value in the security, DevOps, SRE, and cloud engineering job market.

#Conclusion

Congratulations — you've completed the Learn Secret Management with HashiCorp Vault series from episode 0 to episode 25!

In this final episode we wove together an end-to-end enterprise architecture for PT Nusantara Fintech: infrastructure layer (3-node Raft cluster + AWS KMS auto-unseal + TLS), Kubernetes integration (Agent Sidecar Injector + K8s auth in EKS), database security (dynamic PostgreSQL credentials with a 1-hour TTL), application security (Transit for PII encryption), and human access & governance (OIDC via Okta, audit to Grafana Loki, daily snapshots). We closed with a production readiness checklist that turns all the lessons of this series into one gateway to production.

But the most valuable thing isn't the configuration — it's the way of thinking you now have:

1. Secrets must not live in code — all secrets centralized, versioned, and audited. (episode 1)
2. Credentials must be short-lived — dynamic secrets turn a "leak" from a disaster into a minor incident. (episode 5)
3. Keys separated from data — Transit gives encryption without moving keys into the app. (episode 6)
4. Access based on identity, not passwords — OIDC, Kubernetes auth, AppRole; humans and machines authenticated clearly. (episode 10)
5. Everything must be versioned and audited — leases, policies, audit logs, all traceable. (episodes 9, 12, 22)
6. Systems must be available and recoverable — HA, auto-unseal, snapshots, and a trained runbook. (episodes 20, 21, 22)
7. Observability — if it's not measured, it can't be managed. (episode 24)

Your journey doesn't stop here. A few next steps you can take to keep growing:

• Build a real project — take one workload (e.g. a small API), deploy with the architecture in this episode, and drill its full production checklist. Failure drill, rotate keys, revoke leases, restore a snapshot — do all of it.
• Explore the surrounding ecosystem — go deeper on the Vault Secrets Operator (episode 17) for native Kubernetes secret sync, learn Vault Enterprise (Namespaces, Replication, Sentinel) if your organization needs it, or compare OpenBao for full open source needs.
• Deepen layered security — combine with zero-trust concepts, SPIFFE/SPIRE for workload identity, and more formal audit/GRC (PCI-DSS, SOC 2).
• Contribute and share — write Vault policies, Terraform modules for Vault deployment, or share your operational experience in writing. Teaching is the best way to truly master.

Remember the phrase from episode 3: Vault isn't just a secret warehouse — it's a system of trust for the whole organization. Trust that secrets are guarded, access is monitored, and recovery is possible. The engineer who can provide that trust is the engineer the organization relies on.

Thank you for joining this journey to the end. Now — open a terminal, initialize your first Vault cluster, and make secret management something that can be held accountable. Happy building as a true security and infrastructure engineer!