Learning Kubernetes - Episode 41 - Introduction and Explanation of External Secret Manager
In this episode, we'll discuss External Secret Manager like HashiCorp Vault for managing secrets in Kubernetes. We'll learn how to store, retrieve, and rotate secrets securely, and best practices for secret management.
#Introduction
Note
If you want to read the previous episode, you can click the Episode 40 thumbnail below
Episode 40In the previous episode, we explored GitOps, which uses Git as the source of truth for Kubernetes deployments. Now we'll dive into External Secret Manager, which provides secure secret management for Kubernetes applications.
Note: Here I'll be using a Kubernetes Cluster installed through K3s.
Storing secrets in Kubernetes Secrets or Git is insecure. External Secret Manager like HashiCorp Vault provides a centralized, secure way to manage secrets. Think of Vault like a secure vault for your secrets - it encrypts, audits, and controls access to sensitive data. With External Secrets Operator, you can automatically sync secrets from Vault to Kubernetes.
#Understanding External Secret Manager
External Secret Manager is a centralized system for managing, storing, and rotating secrets. It provides encryption, audit logging, and fine-grained access control.
#Why External Secret Manager Matters
1. Centralized Management
All secrets in one secure location.
2. Encryption
Secrets encrypted at rest and in transit.
3. Audit Logging
Track who accessed what and when.
4. Access Control
Fine-grained permissions for secret access.
5. Secret Rotation
Automatically rotate secrets without downtime.
6. Compliance
Meet security compliance requirements.
7. Multi-Environment
Manage secrets across dev, staging, production.
#HashiCorp Vault
Vault is a popular open-source secret management tool.
#Vault Architecture
┌─────────────────────────────────────┐
│ Vault Server │
│ ┌─────────────────────────────┐ │
│ │ Secret Storage │ │
│ │ - Database credentials │ │
│ │ - API keys │ │
│ │ - Certificates │ │
│ └─────────────────────────────┘ │
│ ┌─────────────────────────────┐ │
│ │ Authentication │ │
│ │ - Kubernetes auth │ │
│ │ - JWT auth │ │
│ │ - AppRole │ │
│ └─────────────────────────────┘ │
│ ┌─────────────────────────────┐ │
│ │ Audit Logging │ │
│ │ - Access logs │ │
│ │ - Change history │ │
│ └─────────────────────────────┘ │
└─────────────────────────────────────┘#Installing Vault
# Download Vault
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip
unzip vault_1.15.0_linux_amd64.zip
# Start Vault in dev mode
vault server -dev#Vault Configuration
storage "file" {
path = "/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = false
tls_cert_file = "/vault/tls/vault.crt"
tls_key_file = "/vault/tls/vault.key"
}
ui = true#External Secrets Operator
External Secrets Operator syncs secrets from external systems to Kubernetes.
#Installation
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets-system \
--create-namespace#SecretStore
SecretStore defines how to connect to Vault:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "my-app"#ExternalSecret
ExternalSecret defines which secrets to sync:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: app-secret
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: app-secret
creationPolicy: Owner
data:
- secretKey: database-password
remoteRef:
key: database
property: password
- secretKey: api-key
remoteRef:
key: api
property: key#Vault Secret Engines
#KV Secret Engine
Store key-value secrets:
# Enable KV v2 secret engine
vault secrets enable -version=2 kv
# Store secret
vault kv put kv/database \
username=admin \
password=secret123
# Retrieve secret
vault kv get kv/database#Database Secret Engine
Generate dynamic database credentials:
# Enable database secret engine
vault secrets enable database
# Configure database connection
vault write database/config/my-db \
plugin_name=mysql-database-plugin \
allowed_roles="readonly" \
connection_url="{{username}}:{{password}}@tcp(db.example.com:3306)/" \
username="vault" \
password="vault-password"
# Create role
vault write database/roles/readonly \
db_name=my-db \
creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT SELECT ON *.* TO '{{name}}'@'%';" \
default_ttl="1h" \
max_ttl="24h"
# Generate credentials
vault read database/creds/readonly#PKI Secret Engine
Generate certificates:
# Enable PKI secret engine
vault secrets enable pki
# Generate root certificate
vault write -field=certificate pki/root/generate/internal \
common_name="example.com" \
ttl=87600h > CA_cert.crt
# Create role
vault write pki/roles/example-dot-com \
allowed_domains="example.com" \
allow_subdomains=true \
max_ttl="72h"
# Issue certificate
vault write pki/issue/example-dot-com \
common_name="app.example.com"#Vault Authentication Methods
#Kubernetes Authentication
# Enable Kubernetes auth
vault auth enable kubernetes
# Configure Kubernetes auth
vault write auth/kubernetes/config \
token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
kubernetes_host=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# Create role
vault write auth/kubernetes/role/my-app \
bound_service_account_names=my-app \
bound_service_account_namespaces=default \
policies=my-app-policy \
ttl=24h#AppRole Authentication
# Enable AppRole auth
vault auth enable approle
# Create role
vault write auth/approle/role/my-app \
token_ttl=1h \
token_max_ttl=4h \
policies="my-app-policy"
# Get role ID
vault read auth/approle/role/my-app/role-id
# Generate secret ID
vault write -f auth/approle/role/my-app/secret-id#Practical Examples
#Database Credentials
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: db-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: database
property: username
- secretKey: password
remoteRef:
key: database
property: password
---
apiVersion: v1
kind: Pod
metadata:
name: app-with-db
spec:
containers:
- name: app
image: myapp:latest
env:
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: db-credentials
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-credentials
key: password#API Keys
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-keys
spec:
refreshInterval: 30m
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: api-keys
creationPolicy: Owner
dataFrom:
- extract:
key: api-keys#TLS Certificates
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: tls-cert
spec:
refreshInterval: 24h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: tls-cert
template:
type: kubernetes.io/tls
data:
tls.crt: "{{ .certificate }}"
tls.key: "{{ .private_key }}"
data:
- secretKey: certificate
remoteRef:
key: certificates/app
property: cert
- secretKey: private_key
remoteRef:
key: certificates/app
property: key#Secret Rotation
#Automatic Rotation
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: rotating-secret
spec:
refreshInterval: 1h # Refresh every hour
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: rotating-secret
creationPolicy: Owner
data:
- secretKey: api-key
remoteRef:
key: api-keys/rotating
property: key#Manual Rotation
# Rotate secret in Vault
vault kv put kv/api-keys/rotating \
key=new-api-key-value
# External Secrets will automatically sync
# within the refreshInterval#Common Mistakes and Pitfalls
#Mistake 1: Storing Secrets in Git
Problem: Secrets exposed in Git history.
# DON'T DO THIS - Secrets in Git
apiVersion: v1
kind: Secret
metadata:
name: db-secret
data:
password: c2VjcmV0MTIzSolution: Use External Secret Manager:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-secret
spec:
secretStoreRef:
name: vault-backend
target:
name: db-secret#Mistake 2: Not Rotating Secrets
Problem: Compromised secrets remain active.
Solution: Enable automatic rotation:
spec:
refreshInterval: 1h # Rotate hourly#Mistake 3: Weak Access Control
Problem: Anyone can access all secrets.
Solution: Implement fine-grained policies:
path "kv/data/database" {
capabilities = ["read"]
}
path "kv/data/api-keys" {
capabilities = ["read"]
}
path "kv/data/admin/*" {
capabilities = [] # No access
}#Mistake 4: Not Auditing Secret Access
Problem: No visibility into who accessed secrets.
Solution: Enable audit logging:
vault audit enable file file_path=/vault/logs/audit.log#Mistake 5: Hardcoding Secrets in Code
Problem: Secrets exposed in source code.
Solution: Always use secret management:
# DON'T DO THIS
export DB_PASSWORD="secret123"export DB_PASSWORD=$(kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d)#Best Practices
#1. Use Kubernetes Authentication
auth:
kubernetes:
mountPath: "kubernetes"
role: "my-app"#2. Set Appropriate TTLs
vault write auth/kubernetes/role/my-app \
ttl=24h \
max_ttl=7d#3. Implement Least Privilege
# Only grant necessary permissions
path "kv/data/my-app/*" {
capabilities = ["read"]
}#4. Enable Audit Logging
vault audit enable file file_path=/vault/logs/audit.log#5. Rotate Secrets Regularly
spec:
refreshInterval: 1h#6. Use Separate Vaults for Environments
Vault Dev
Vault Staging
Vault Production#7. Backup Vault Data
vault operator raft snapshot save vault-backup.snap#8. Monitor Secret Access
# Review audit logs
vault audit list
tail -f /vault/logs/audit.log#Vault vs Kubernetes Secrets
| Aspect | Vault | Kubernetes Secrets |
|---|---|---|
| Encryption | Yes | Optional |
| Audit Logging | Yes | Limited |
| Access Control | Fine-grained | RBAC only |
| Secret Rotation | Automatic | Manual |
| Centralized | Yes | Per-cluster |
| Compliance | Yes | Limited |
| Cost | Self-hosted or managed | Free |
#Monitoring and Troubleshooting
#Check Secret Sync Status
kubectl describe externalsecret app-secret
kubectl get externalsecret -o wide#View Vault Audit Logs
vault audit list
tail -f /vault/logs/audit.log#Test Vault Connectivity
kubectl exec -it pod-name -- \
curl -k https://vault.example.com:8200/v1/sys/health#Debug External Secrets
kubectl logs -n external-secrets-system \
deployment/external-secrets#Conclusion
In episode 41, we've explored External Secret Manager in Kubernetes in depth. We've learned how to use HashiCorp Vault with External Secrets Operator to securely manage secrets.
Key takeaways:
- External Secret Manager provides centralized secret management
- HashiCorp Vault - Popular open-source secret manager
- External Secrets Operator - Syncs secrets from Vault to Kubernetes
- SecretStore - Defines connection to Vault
- ExternalSecret - Defines which secrets to sync
- Secret Engines - KV, Database, PKI, etc.
- Authentication Methods - Kubernetes auth, AppRole, JWT
- Encryption - Secrets encrypted at rest and in transit
- Audit Logging - Track all secret access
- Access Control - Fine-grained permissions
- Secret Rotation - Automatic rotation without downtime
- Never store secrets in Git - Use external manager
- Enable audit logging - Monitor secret access
- Implement least privilege - Only grant necessary permissions
- Rotate secrets regularly - Minimize compromise window
External Secret Manager is essential for production Kubernetes deployments to ensure secrets are secure, auditable, and compliant.
Note
If you want to continue to the next episode, you can click the Episode 42 thumbnail below
Episode 42