Learn GitOps with ArgoCD - Secrets Management

#Introduction

In episode 11 we automated Application creation with ApplicationSet. All those manifests — Application, Deployment, ConfigMap — we dare to store in Git because they're safe to read publicly. But there's one type of resource that must not be: Secret. Database passwords, API tokens, TLS keys, cloud credentials — if stored raw in Git, one leaked repository means the whole infrastructure is exposed. In this episode we discuss how to store secrets safely in GitOps, from simple to enterprise.

Why does this matter? "Never commit secrets" is rule number one of GitOps, but in practice many teams break it because they don't know the alternatives. Yet the solutions are mature and varied: Sealed Secrets for encryption in Git, the External Secrets Operator for syncing from external storage, and SOPS for file encryption. Understanding when to use which is a mandatory skill for platform engineers.

#The Secret Challenge in GitOps

GitOps demands everything lives in Git — but secrets are exactly what's most dangerous in Git. Three fundamental problems:

1. Permanent history — once committed, a secret is recorded in Git forever, even after it's removed in a later commit.
2. Spreading access — every repository clone carries the secret to every developer machine.
3. Difficult rotation — if a secret leaks, replacing it means rewriting history, not just editing.

The basic principle: never store raw secrets, only encrypted representations or references to another source.

#Solution Map

#Sealed Secrets (Bitnami)

The most popular approach to start with: the secret is encrypted before it enters Git, and only a controller inside the cluster holds the private key to decrypt it.

#Installation

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system \
  --set-string fullnameOverride=sealed-secrets-controller \
  sealed-secrets/sealed-secrets
kubeseal --version

The controller creates a key pair: the private key is stored as a Secret in the cluster (never leaves it), the public key is exported for the sealing process.

#Sealing a Secret

secret.yaml (raw, only temporary)

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: billing-prod
type: Opaque
stringData:
  DB_PASSWORD: s3cr3t-rahasia

kubeseal --format yaml < secret.yaml > sealed-secret.yaml
kubectl apply -f sealed-secret.yaml
cat sealed-secret.yaml

The resulting sealed-secret.yaml file is safe to commit to Git — its content is encrypted and can only be opened by the controller in the cluster holding the private key. The controller decrypts it and creates the real db-credentials Secret in the destination namespace.

#Committing the SealedSecret to Git

Application for the SealedSecret

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: billing-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/devnull/billing-repo.git
    path: secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: billing-prod

#External Secrets Operator (ESO)

In contrast, ESO doesn't store secrets in Git at all — Git only holds a reference called an ExternalSecret, and the operator syncs the actual values from external storage.

#Installation

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets -n external-secrets \
  external-secrets/external-secrets --create-namespace

#SecretStore

SecretStore defines the connection to a backend — Vault, AWS Secrets Manager, GCP Secret Manager, and dozens of other providers:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: billing-prod
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret/data/billing
      auth:
        kubernetes:
          mountPath: kubernetes
          role: billing-role

#ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: billing-prod
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: billing/db
        property: password

ESO reads the value from Vault and creates the db-credentials Secret in the cluster. What gets committed to Git is only the SecretStore and ExternalSecret — no secret values. Bonus: rotation only happens in Vault, and ESO syncs automatically according to refreshInterval.

#SOPS (Mozilla)

SOPS encrypts files (YAML, JSON, env) directly in Git using GPG or age keys:

sops -e -i secrets/values.prod.yaml
sops -d secrets/values.prod.yaml
git add secrets/ && git commit -m "chore: encrypt prod values"

The encrypted file remains YAML so diffs between commits are still readable. Integration with ArgoCD is usually through a plugin (SOPS encrypt/decrypt on the repo-server) or by using kustomize + generators. SOPS is a good choice when the team wants values to stay "in Git" but secure.

#Vault & Cloud Secret Managers

For enterprise scale, secrets are placed in centralized storage: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault. The values never touch Git — Git only stores references, and ESO (or Vault's template engine) connects the two. This model gives a single source of truth for secrets and a clear audit trail.

#Best Practices

• Never commit plain secrets — from day one; a secret that has ever entered history is considered compromised.
• Separate secrets from manifests — keep secrets in a separate directory or repository, with strict Git-side RBAC.
• RBAC for secrets — restrict who can read Secrets in the cluster (Kubernetes RBAC) and who can open the backend.
• Regular rotation — set a rotation schedule; with ESO, rotating in the backend syncs automatically.
• Audit — enable audit logs in Vault/cloud managers and monitor Secret access in the cluster.

#Common Pitfalls

1. Committing raw secrets "for now, encrypted later". There is no "later" — once it enters Git, treat it as compromised and rotate it.
2. No backup of the Sealed Secrets keys. A lost private key = none of the SealedSecrets can be opened. Back up the key Secret somewhere safe.
3. Sealed Secrets keys leaking to CI. Don't export the private key for pipelines; only the public key.
4. ExternalSecret targeting the wrong namespace. Make sure target.name and the namespace match what the Deployment needs.
5. Manual rotation without synchronization. If rotation only happens in the backend but refreshInterval never fires, the Secret in the cluster goes stale.

#Closing

This episode answered the classic GitOps dilemma: how to store secrets without leaking them into Git. We compared Sealed Secrets (encryption in Git), the External Secrets Operator (reference + sync from a backend), SOPS (file encryption), and Vault and cloud secret managers as a centralized source, closing with best practices and common pitfalls.

The points you should take with you:

• Never commit raw secrets — once leaked, forever leaked.
• Sealed Secrets fits when you want secrets to stay "in Git" in encrypted form.
• ESO fits when you want secrets sourced from Vault or a cloud secret manager.
• SOPS is ideal for encrypting config files whose diffs remain readable.
• Rotation and audit are part of secret management, not optional.

Deployments are now safe and contain properly stored secrets. But there's one dimension we haven't controlled yet: the order and side effects during sync. In the next episode 13 we discuss Resource Hooks & Lifecycle: PreSync, Sync, PostSync, and SyncFail hooks for database migrations, smoke tests, and rollbacks — plus controlling execution order with sync waves. See you in episode 13!