HashiCorp Vault for Secret Management in Production Systems

#Introduction

Secrets are everywhere in production systems—database passwords, API keys, TLS certificates, encryption keys. Yet most teams still manage them poorly: hardcoded in repositories, scattered across config files, or stored in plaintext environment variables.

This is where HashiCorp Vault comes in. It's not just a secret storage system; it's a centralized secret management platform that handles credential generation, rotation, and audit trails automatically.

In this guide, we'll explore how Vault works, why it matters for production systems, and how to implement it properly. Whether you're running Kubernetes, traditional VMs, or hybrid infrastructure, Vault provides a unified approach to secret management that scales.

#Understanding the Secret Management Problem

Before diving into Vault, let's understand why secret management is hard:

The traditional approach breaks down because:

• Secrets in code repositories are exposed to anyone with access
• Rotating secrets requires redeploying applications
• There's no audit trail of who accessed what secret
• Different teams use different secret storage methods
• Secrets leak through logs, error messages, and backups
• Static credentials can't be revoked instantly

Vault solves these problems by centralizing secret management and automating credential lifecycle.

#How Vault Works: Core Concepts

#Authentication Methods

Before accessing secrets, clients must authenticate to Vault. Vault supports multiple auth methods:

• Token Auth: Direct token-based access (useful for testing)
• AppRole: Machine-to-machine authentication
• Kubernetes: Native Kubernetes pod authentication
• JWT/OIDC: Identity provider integration
• AWS IAM: AWS service authentication
• LDAP/Active Directory: Enterprise directory integration

Think of auth methods as different ways to prove "I am who I say I am" to Vault.

#Secrets Engines

Secrets engines are Vault's backend systems that generate, store, or manage secrets:

• KV (Key-Value): Simple secret storage
• Database: Dynamic database credentials
• PKI: Certificate generation and management
• SSH: SSH key management
• AWS: Dynamic AWS credentials
• Kubernetes: Dynamic Kubernetes service account tokens

The key difference: static secrets (stored once) vs. dynamic secrets (generated on-demand with automatic rotation).

#Policies

Policies define what authenticated clients can do. They follow a path-based access control model:

path "secret/data/app/*" {
  capabilities = ["read", "list"]
}
 
path "database/creds/app-role" {
  capabilities = ["read"]
}
 
path "auth/token/renew-self" {
  capabilities = ["update"]
}

Policies are the principle of least privilege in action—each application gets access only to the secrets it needs.

#Setting Up Vault in Production

#Installation & Initialization

# Download Vault
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip
unzip vault_1.15.0_linux_amd64.zip
sudo mv vault /usr/local/bin/
 
# Verify installation
vault version

Initialize Vault (generates unseal keys and root token):

vault operator init \
  -key-shares=5 \
  -key-threshold=3

This creates 5 unseal keys where any 3 can unseal Vault. Store these securely—they're critical for recovery.

#Unsealing Vault

Vault starts in a sealed state. Unsealing requires threshold number of unseal keys:

vault operator unseal <unseal-key-1>
vault operator unseal <unseal-key-2>
vault operator unseal <unseal-key-3>

After threshold keys are provided, Vault unseals and becomes operational.

#Configuration: Storage Backend

Vault needs persistent storage. Common options:

storage "raft" {
  path = "/opt/vault/data"
  node_id = "vault-1"
}
 
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file = "/opt/vault/tls/vault.key"
}
 
api_addr = "https://vault.example.com:8200"
cluster_addr = "https://vault-1.example.com:8201"
ui = true

For production, use Integrated Storage (Raft) for high availability or external backends like Consul, S3, or PostgreSQL.

#Practical Implementation: Database Credentials

One of Vault's most powerful features is dynamic database credentials. Instead of static passwords, Vault generates temporary credentials on-demand.

#Configure Database Secret Engine

vault secrets enable database

Configure connection to your database:

vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.example.com:5432/postgres" \
  username="vault_admin" \
  password="vault_admin_password"

Define a role that generates credentials:

vault write database/roles/app-role \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Now applications request credentials:

vault read database/creds/app-role

Response:

{
  "lease_id": "database/creds/app-role/abc123",
  "lease_duration": 3600,
  "data": {
    "username": "v-token-app-role-abc123",
    "password": "A1b2C3d4E5f6G7h8I9j0"
  }
}

The credentials are temporary, automatically rotated, and revoked when the lease expires.

#Integration with Kubernetes

Vault integrates natively with Kubernetes through the Kubernetes auth method:

#Enable Kubernetes Auth

vault auth enable kubernetes

Configure Kubernetes connection:

vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

Create a role for your application:

vault write auth/kubernetes/role/app-role \
  bound_service_account_names=app \
  bound_service_account_namespaces=production \
  policies="app-policy" \
  ttl=1h

#Pod Integration with Vault Agent

Deploy Vault Agent as a sidecar to inject secrets:

Pod with Vault Agent Sidecar

Copy

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  serviceAccountName: app
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: vault-token
      mountPath: /vault/secrets
  - name: vault-agent
    image: vault:latest
    args:
    - agent
    - -config=/vault/config/agent.hcl
    volumeMounts:
    - name: vault-config
      mountPath: /vault/config
    - name: vault-token
      mountPath: /vault/secrets
  volumes:
  - name: vault-config
    configMap:
      name: vault-agent-config
  - name: vault-token
    emptyDir: {}

Vault Agent configuration:

vault {
  address = "https://vault.vault.svc.cluster.local:8200"
}
 
auto_auth {
  method {
    type = "kubernetes"
    config = {
      role = "app-role"
    }
  }
  sink {
    type = "file"
    config = {
      path = "/vault/secrets/.vault-token"
    }
  }
}
 
template {
  source = "/vault/config/app-config.tpl"
  destination = "/vault/secrets/app-config.json"
}

#Secret Rotation & Lifecycle Management

#Automatic Secret Rotation

For database credentials, Vault handles rotation automatically:

vault write database/roles/app-role \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}';" \
  rotation_statements="ALTER ROLE \"{{name}}\" WITH PASSWORD '{{password}}';" \
  default_ttl="1h" \
  max_ttl="24h"

Vault rotates credentials before expiration, ensuring applications always have valid credentials.

#Manual Secret Rotation

For static secrets, implement rotation policies:

# Generate new key
NEW_KEY=$(openssl rand -hex 32)
 
# Update in Vault
vault kv put secret/api-keys/external-service key=$NEW_KEY
 
# Update external service
curl -X POST https://api.example.com/keys \
  -H "Authorization: Bearer $OLD_KEY" \
  -d "{\"new_key\": \"$NEW_KEY\"}"
 
# Revoke old key
curl -X DELETE https://api.example.com/keys/$OLD_KEY \
  -H "Authorization: Bearer $NEW_KEY"

#Audit & Compliance

Vault maintains comprehensive audit logs of all secret access:

vault audit enable file file_path=/var/log/vault-audit.log

Audit logs capture:

• Who accessed what secret
• When the access occurred
• Whether the request succeeded or failed
• Request and response metadata

Example audit log entry:

{
  "time": "2026-02-10T14:32:15.123456Z",
  "type": "request",
  "auth": {
    "client_token": "s.xxxxxxxxxxxxxxxx",
    "accessor": "kKz7xWHYzxxx",
    "display_name": "kubernetes-app",
    "policies": ["app-policy"],
    "token_ttl": 3600,
    "token_type": "service"
  },
  "request": {
    "id": "abc-123-def",
    "operation": "read",
    "path": "database/creds/app-role",
    "data": {}
  },
  "response": {
    "data": {
      "username": "v-token-app-role-abc123",
      "password": "***"
    }
  }
}

This audit trail is essential for compliance (SOC 2, PCI-DSS, HIPAA) and security investigations.

#Common Mistakes & Pitfalls

#Mistake 1: Storing Unseal Keys Insecurely

The problem: Unseal keys are stored in plaintext or in the same location as Vault.

Why it happens: Teams rush deployment and skip key management.

How to avoid it:

• Use Shamir key sharing (distribute keys to different people)
• Store keys in a secure vault (physical safe, HSM, or encrypted storage)
• Never store all keys in one location
• Consider auto-unseal with cloud KMS for production

#Mistake 2: Not Rotating Secrets

The problem: Secrets remain static for months or years.

Why it happens: Manual rotation is tedious; teams forget or deprioritize it.

How to avoid it:

• Enable automatic rotation for database credentials
• Set up scheduled rotation jobs for static secrets
• Use Vault's lease system to enforce TTLs
• Monitor rotation failures with alerts

#Mistake 3: Overly Permissive Policies

The problem: Applications have access to secrets they don't need.

Why it happens: Teams use wildcard policies for convenience.

How to avoid it:

• Follow principle of least privilege strictly
• Use specific paths, not wildcards
• Audit policies regularly
• Test policies before deployment

#Mistake 4: Ignoring High Availability

The problem: Single Vault instance becomes a single point of failure.

Why it happens: HA setup is complex; teams skip it for "non-critical" environments.

How to avoid it:

• Deploy Vault in HA mode from the start
• Use Integrated Storage (Raft) for clustering
• Set up load balancing across Vault nodes
• Test failover scenarios

#Mistake 5: Not Monitoring Vault Health

The problem: Vault issues go unnoticed until applications fail.

Why it happens: Vault is treated as "set and forget" infrastructure.

How to avoid it:

• Monitor Vault's health endpoint
• Alert on seal status changes
• Track lease expirations
• Monitor audit log volume

#Best Practices for Production

#1. High Availability Setup

Deploy Vault in HA mode with multiple nodes:

storage "raft" {
  path = "/opt/vault/data"
  node_id = "vault-1"
  retry_join {
    leader_api_addr = "https://vault-2.example.com:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-3.example.com:8200"
  }
}
 
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file = "/opt/vault/tls/vault.key"
}
 
api_addr = "https://vault-1.example.com:8200"
cluster_addr = "https://vault-1.example.com:8201"

#2. TLS Everywhere

Always use TLS for Vault communication:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout /opt/vault/tls/vault.key \
  -out /opt/vault/tls/vault.crt \
  -subj "/CN=vault.example.com"
 
chmod 600 /opt/vault/tls/vault.key

#3. Separate Auth & Data Paths

Use different auth methods for different workloads:

# Kubernetes pods
vault auth enable kubernetes
 
# CI/CD pipelines
vault auth enable jwt
 
# Human operators
vault auth enable oidc
 
# Service-to-service
vault auth enable approle

#4. Implement Lease Management

Set appropriate TTLs for different secret types:

# Short-lived credentials for sensitive operations
vault write database/roles/sensitive-role \
  default_ttl="15m" \
  max_ttl="1h"
 
# Longer TTL for less sensitive operations
vault write database/roles/app-role \
  default_ttl="24h" \
  max_ttl="720h"

#5. Backup & Disaster Recovery

Regularly backup Vault data:

vault operator raft snapshot save vault-backup-$(date +%Y%m%d).snap

Store backups securely and test restoration regularly.

#When NOT to Use Vault

Vault is powerful but not always the right tool:

• Simple projects with few secrets: Vault adds operational complexity. For small projects, environment variables or simple config files might suffice.
• Extremely latency-sensitive applications: Vault adds network round-trips. Cache secrets locally when possible.
• Teams without DevOps expertise: Vault requires operational knowledge. Ensure your team can maintain it.
• Fully managed secret services: If using AWS Secrets Manager, Azure Key Vault, or Google Secret Manager, you might not need Vault. Evaluate based on multi-cloud requirements.

#Conclusion

HashiCorp Vault transforms secret management from a security liability into a controlled, auditable process. By centralizing secrets, automating rotation, and providing comprehensive audit trails, Vault enables teams to meet compliance requirements while improving security posture.

The key takeaways:

• Vault provides centralized secret management with dynamic credential generation
• Integrate Vault with your infrastructure (Kubernetes, databases, cloud services)
• Implement proper authentication, policies, and audit logging from day one
• Deploy in HA mode with TLS for production reliability
• Automate secret rotation to reduce manual overhead
• Monitor Vault health and audit logs continuously

Start with a pilot deployment in a non-critical environment, understand the operational model, then expand to production. The investment in proper secret management pays dividends in security, compliance, and operational peace of mind.