Learning Kubernetes - Episode 21.2

#Introduction

Note

If you want to read the previous episode, you can click the Episode 21.1 thumbnail below

In the previous episode, we deep-dived into PersistentVolume and PersistentVolumeClaim for durable storage. In this episode (21.2), we'll focus on Secrets — Kubernetes' native object for managing sensitive data such as database passwords, API tokens, TLS certificates, and SSH keys.

Note: Here I'll be using a Kubernetes Cluster installed through K3s.

Most real-world applications need credentials of some kind. Without a proper mechanism, developers end up hardcoding secrets in container images or environment variable lists, creating security liabilities. Kubernetes Secrets give you a structured, API-driven way to separate sensitive data from application code and configuration.

#What Is a Secret?

A Secret is a Kubernetes API object that stores a small amount of sensitive data in key-value pairs. Secrets are stored in etcd and can be consumed by Pods as environment variables, files via volumes, or used by the kubelet to pull private container images.

Think of Secrets like a locked safe that only authorized people can open. The application doesn't carry its own password vault — it requests access to the vault at runtime.

Key characteristics of Secrets:

Namespace-scoped - A Secret belongs to one Namespace
Base64-encoded - Values are base64-encoded (not encrypted by default)
Multiple types - Generic, TLS, Docker registry credentials, service account tokens
Multiple consumption patterns - Environment variables, volume mounts, image pull secrets
RBAC-controlled - Access governed by Kubernetes RBAC
Encryption at rest - Optional, must be configured explicitly via EncryptionConfiguration

Warning

Kubernetes Secrets are base64-encoded by default, not encrypted. Anyone with kubectl get secret access or read access to etcd can decode them trivially. Always enable encryption at rest and use strict RBAC in production.

#Secret vs ConfigMap

Both Secrets and ConfigMaps store key-value data. The difference is intent and handling:

Feature	Secret	ConfigMap
Intended for	Sensitive data	Non-sensitive config
Storage format	Base64-encoded	Plain text
Encryption at rest	Supported (opt-in)	Not typically encrypted
Mounted as `tmpfs`	Yes (in-memory)	No
RBAC granularity	`secrets` resource	`configmaps` resource
Visible in `kubectl describe pod`	Values hidden	Values shown

Use ConfigMaps for application configuration. Use Secrets for anything that would be harmful if leaked: passwords, tokens, certificates, private keys.

#Secret Types

Kubernetes supports several built-in Secret types, each designed for a specific use case.

#Opaque (Generic)

The default type for arbitrary user-defined data.

Or use stringData to provide values in plain text (Kubernetes encodes them automatically):

Tip

Prefer stringData in your manifests — it's more readable and avoids manual base64 encoding mistakes. Kubernetes will automatically convert the values to base64 when storing them.

#kubernetes.io/tls

Used for TLS certificates and private keys. Consumed by Ingress controllers and other TLS-aware components.

Creating TLS secrets from files is more common:

#kubernetes.io/dockerconfigjson

Used for authenticating to private container image registries.

Reference it in a Pod spec:

#kubernetes.io/service-account-token

Automatically created by Kubernetes for ServiceAccounts. Used by Pods to authenticate to the API server. Modern clusters use projected service account tokens instead.

#Summary of Built-in Secret Types

Type	Use Case
`Opaque`	Generic key-value secrets
`kubernetes.io/tls`	TLS certificates and keys
`kubernetes.io/dockerconfigjson`	Registry authentication
`kubernetes.io/basic-auth`	Username/password basic auth
`kubernetes.io/ssh-auth`	SSH private keys
`kubernetes.io/service-account-token`	ServiceAccount tokens
`bootstrap.kubernetes.io/token`	Node bootstrap tokens

#Creating Secrets

#Using kubectl (Imperative)

# From literal values
sudo kubectl create secret generic db-credentials \
    --from-literal=username=admin \
    --from-literal=password=supersecret
 
# From files
sudo kubectl create secret generic app-certs \
    --from-file=ca.crt \
    --from-file=server.crt \
    --from-file=server.key

# List secrets
sudo kubectl get secrets
 
# Describe (values are hidden)
sudo kubectl describe secret db-credentials
 
# Get raw secret (base64)
sudo kubectl get secret db-credentials -o yaml
 
# Decode a specific key
sudo kubectl get secret db-credentials \
    -o jsonpath='{.data.password}' | base64 -d

#Using YAML Manifests (Declarative)

Caution

Never commit Secret manifests with real values to version control. Your git history is permanent. Use sealed-secrets, External Secrets Operator, or Vault to manage secrets as code safely. More on this in the best practices section.

#Consuming Secrets in Pods

Secrets can be consumed in two ways: environment variables or volume mounts. Each has distinct trade-offs.

#As Environment Variables

The simplest pattern — directly inject secret values as container environment variables.

#Inject a Specific Key

#Inject All Keys from a Secret

This mounts every key in db-credentials as an environment variable. The key names become the variable names.

Warning

Environment variables are visible in process listings (/proc/<pid>/environ) and may be logged by third-party libraries or crash reports. For maximum security, prefer volume mounts for sensitive credentials.

#As Volume Mounts (Files)

Secrets mounted as volumes are stored on tmpfs (in-memory filesystem), which is safer than disk. Each key becomes a file inside the mount directory.

Inside the container, the files will be:

bash

/etc/secrets/DB_HOST
/etc/secrets/DB_PORT
/etc/secrets/DB_NAME
/etc/secrets/DB_USER
/etc/secrets/DB_PASSWORD

#Mount Specific Keys Only

This mounts only the specified keys, with custom file names and permissions.

#Comparing Env Vars vs Volume Mounts

Aspect	Environment Variable	Volume Mount
Visibility in process	`cat /proc/self/environ`	Not exposed
Storage	Memory (env)	`tmpfs` (in-memory)
Dynamic updates	Requires Pod restart	Updated without restart
File permissions	N/A	Configurable (`mode`)
Audit trail	Harder to audit	Easier (file access logging)
Best for	Simple apps, legacy compat	Production, security-sensitive

#Practical Implementation

#Example 1: Database Application with Secret

A typical pattern: inject database credentials securely into a web application.

apiVersion: v1
kind: Secret
metadata:
    name: postgres-credentials
    namespace: production
type: Opaque
stringData:
    POSTGRES_USER: appuser
    POSTGRES_PASSWORD: Sup3rS3cur3P@ss!
    POSTGRES_DB: myappdb

apiVersion: apps/v1
kind: Deployment
metadata:
    name: web-app
    namespace: production
spec:
    replicas: 2
    selector:
        matchLabels:
            app: web-app
    template:
        metadata:
            labels:
                app: web-app
        spec:
            containers:
                - name: app
                  image: myapp:latest
                  ports:
                      - containerPort: 8080
                  env:
                      # Non-sensitive config from ConfigMap
                      - name: APP_ENV
                        value: production
                      - name: LOG_LEVEL
                        value: info
                  # Sensitive config from Secret
                  envFrom:
                      - secretRef:
                            name: postgres-credentials
                  volumeMounts:
                      - name: tls-certs
                        mountPath: /etc/ssl/app
                        readOnly: true
            volumes:
                - name: tls-certs
                  secret:
                      secretName: tls-secret
                      defaultMode: 0400

#Example 2: TLS Termination with Ingress

#Example 3: Private Registry Pull Secret

Attaching the pull secret to the ServiceAccount means all Pods using this ServiceAccount automatically get access to the private registry — no need to specify imagePullSecrets on every Pod.

#Example 4: Updating a Secret

Note

Pods consuming Secrets as volume mounts will see the updated secret within ~1 minute (kubelet sync period) without restart. Pods consuming Secrets as environment variables require a restart to pick up the new values.

#Secret Encryption at Rest

By default, Kubernetes stores Secrets in etcd as base64-encoded plain text. Anyone with etcd access can read them. To enable true encryption at rest, configure an EncryptionConfiguration.

Apply this by passing --encryption-provider-config flag to the API server. After enabling, encrypt existing secrets:

Important

Managed Kubernetes services (GKE, EKS, AKS) typically provide encryption at rest for Secrets by default using their cloud KMS. Always verify this is enabled in your cloud provider's documentation.

#Common Mistakes and Pitfalls

#Mistake 1: Storing Secrets in Git

Problem: Committing Secret YAML files with real values to version control. The values persist in git history even after deletion.

Solution: Use one of these approaches:

Sealed Secrets (Bitnami) — encrypt secrets client-side, safe to commit
External Secrets Operator — sync secrets from Vault, AWS SSM, GCP Secret Manager
SOPS — encrypt YAML/JSON files using age or KMS

#Mistake 2: Overly Permissive RBAC

Problem: Granting broad get secrets access exposes all secrets in the Namespace.

Solution: Follow least privilege. Grant access only to specific secrets:

#Mistake 3: Using Environment Variables for Highly Sensitive Data

Problem: Environment variables are visible to all processes in the container and may be logged.

Solution: Mount sensitive data as volume files with restrictive permissions:

#Mistake 4: Not Setting File Permissions

Problem: Secret files mounted in volumes are readable by all users in the container.

Solution: Set restrictive defaultMode and per-item mode:

#Mistake 5: Not Rotating Secrets

Problem: Static, long-lived credentials are a liability. A leaked credential remains valid indefinitely.

Solution: Implement secret rotation. Use short-lived tokens where possible (e.g., Vault dynamic secrets, AWS IAM roles for service accounts).

#Best Practices

#Use External Secret Management in Production

Don't manage secrets natively in Kubernetes at scale. Use an external secrets backend:

#Enable Encryption at Rest

Always enable EncryptionConfiguration pointing to a KMS provider (not just aescbc with a static key) in production.

#Apply Strict RBAC

Restrict who can list and get secrets:

Tip

Removing the list verb prevents users from enumerating all Secret names in a Namespace — a useful defense-in-depth measure even if they can get specific secrets by name.

#Mount Secrets as Read-Only

Always add readOnly: true to Secret volume mounts:

#Audit Secret Access

Enable Kubernetes audit logging with a policy that captures get and watch operations on Secrets:

#Use Immutable Secrets

Mark Secrets as immutable to prevent accidental modification and reduce API server load (Kubernetes stops watching for changes):

#When NOT to Use Kubernetes Secrets

Kubernetes Secrets are convenient but not always the right tool:

Large binary payloads — Secrets have a 1 MiB size limit. Use object storage (S3, GCS) for large files.
Secrets shared across many clusters — Managing the same secret per-cluster is operationally expensive. Use a centralized vault (HashiCorp Vault, AWS Secrets Manager) as the single source of truth.
Secrets requiring dynamic generation — If credentials need to be generated on-demand per-pod (e.g., short-lived database credentials), Vault's dynamic secrets are purpose-built for this.
Compliance-regulated environments — Some compliance frameworks (PCI-DSS, HIPAA) require dedicated secrets management hardware or services. Kubernetes Secrets alone may not satisfy the audit requirements.

Note

Kubernetes Secrets are fine for simple deployments and development. For production at scale, always layer them with proper encryption, external secrets management, and strict RBAC.

#Conclusion

In episode 21.2, we've covered Kubernetes Secrets comprehensively — from what they are and their types, to how to create and consume them, to securing them properly in production.

Key takeaways:

Secrets store sensitive data as base64-encoded key-value pairs in etcd
Base64 is encoding, not encryption — always enable encryption at rest
Common Secret types: Opaque, kubernetes.io/tls, kubernetes.io/dockerconfigjson
Secrets can be consumed as environment variables (simpler) or volume mounts (more secure)
Volume-mounted secrets are stored on tmpfs (in-memory), more secure than env vars
Volume mounts update dynamically (~1 min); env vars require Pod restart
Never commit Secret manifests with real values to version control
Use RBAC to restrict who can access which secrets — avoid broad list permissions
Use External Secrets Operator, Sealed Secrets, or HashiCorp Vault for production-grade secrets management
Set immutable: true for secrets that don't change to reduce API server overhead
Enable Kubernetes audit logging to track Secret access

Proper secrets management is one of the most security-critical aspects of running Kubernetes in production. Getting it right from the start saves painful security incidents later.

Are you getting a clearer picture of how Kubernetes manages sensitive data? Keep the momentum going and look forward to the next episode!

Note

If you want to continue to the next episode, you can click the Episode 22 thumbnail below

Episode 22

Learning Kubernetes - Episode 21.2 - Secret

Related Posts