Learn PKI - Monitoring, Observability & Automation Pipeline

#Introduction

Episode 18 built a production-ready multi-CA architecture: HA for step-ca, leader election for cert-manager, and recovery playbooks. But one question gapes open: how do we know everything is healthy? An expired certificate is usually only noticed when a TLS error appears in the middle of peak hours. No amount of infrastructure helps if no one is watching it.

Episode 19 closes that gap with two things: observability and automation. First, we learn how to monitor certificate validity, read cert-manager status, expose step-ca metrics to Prometheus, and assemble alerting that genuinely helps. Second, we build an automated pipeline — from Terraform and Helm, through GitOps with ArgoCD or Flux, to a pipeline that issues and deploys certificates without manual touch.

#Expiry Monitoring with openssl x509

The simplest way to monitor certificate validity is openssl x509 -checkend, which returns a success status if the certificate is still alive within a certain number of seconds. This command can be looped over an entire certificate directory so we have a simple sensor with no dependencies.

for cert in /etc/ssl/certs/*.pem; do
  openssl x509 -checkend 86400 -noout -in "$cert" || echo "expiring soon: $cert"
done

Change the 86400 number according to tolerance: 86400 seconds for one day, 604800 for seven days. A cron or systemd timer runs this script every morning, and non-empty output is sent to Slack or email. For a quick single-certificate check, openssl x509 -enddate -noout -in cert.pem displays the expiry date directly.

#Status and Conditions in cert-manager

If you operate on Kubernetes, cert-manager already provides rich status. Every Certificate object has conditions explaining whether the certificate is ready to use, being processed, or failed. Conditions such as Ready and Issuing, along with reasons such as CertificateRequestFailed, give hints about the root cause without manually opening logs.

kubectl get certificate -A
kubectl get certificaterequest -A
kubectl describe certificate app-cert

kubectl describe shows the complete conditions, including last transition time and the controller's message. A good habit: save this output to a long-term logging system, then build a dashboard from cert-manager metrics. That way, a certificate that fails to renew is visible from the graph before users report it.

#step-ca Metrics for Prometheus

step-ca exposes Prometheus metrics at a dedicated endpoint that can be enabled at startup. These metrics include the number of issued certificates, process duration, and server health status. Put this endpoint behind a ServiceMonitor if you use kube-prometheus-stack, or scrape directly if Prometheus runs outside Kubernetes.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: step-ca
spec:
  endpoints:
    - port: metrics
      path: /metrics
  selector:
    matchLabels:
      app: step-ca

The same metrics can also be fetched with a manual curl for quick debugging — for example curl -s localhost:9000/metrics to see a metrics snapshot from a misbehaving instance. The key to observability is not just having metrics, but having the habit of using them.

#Alerting Done Right

Metrics without alerting only end up on a dashboard nobody opens. Good alerting rules focus on things that need action: certificates nearing expiry, suspicious issuance rate spikes, and unhealthy CAs. Avoid noisy alerts — an alert that always fires will be ignored.

groups:
  - name: pki
    rules:
      - alert: CertificateExpiringSoon
        expr: certificate_expires_in_seconds < 604800
        for: 1h
        labels:
          severity: warning

Define alert ownership from the start: who is contacted for application certificates, who for the root CA, who for policy. Alertmanager routing can separate the three into different channels, so on-call is not drowned by notifications outside their responsibility.

#Automation with Terraform

Terraform brings PKI into the Infrastructure as Code flow. The ACME provider for Terraform lets you issue certificates as resources, complete with automatic renewal lifecycle. This is ideal for classic infrastructure that is not yet fully Kubernetes, or for certificates that must be guaranteed to exist before other services are provisioned.

resource "acme_registration" "admin" {
  account_key_pem = tls_private_key.reg.private_key_pem
  email_address   = "ops@example.com"
}
 
resource "acme_certificate" "web" {
  account_key_pem           = acme_registration.admin.account_key_pem
  common_name               = "app.example.com"
  dns_challenge {
    provider = "cloudflare"
  }
}

Terraform can also manage the cert-manager and step-ca Helm releases, so their versions, values, and issuer configuration are recorded as reviewable code. Terraform state is a critical asset — store it in a remote backend with locking, not on a laptop.

#GitOps with ArgoCD and Flux

For Kubernetes clusters, GitOps places the git repo as the source of truth. ArgoCD and Flux continuously compare cluster state with what is in the repo and fix any drift. This also applies to certificates: Certificate, Issuer, and ClusterIssuer are declared in git, then applied automatically to the cluster.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: pki
spec:
  sourceRef:
    kind: GitRepository
    name: infra
  path: ./pki/production
  prune: true

The advantages are huge: every issuer or certificate policy change goes through a pull request, is reviewed, then applied. If a configuration is wrong, GitOps shows the drift and makes rollback to the previous version easy. Your PKI now has an automatic audit trail without extra tools.

#Issue and Deploy Pipeline

The final step is connecting everything: when a new service needs its certificate, the pipeline issues, stores, and deploys that certificate in sequence without a human in the middle. For non-Kubernetes environments, a simple script suffices; for Kubernetes, let cert-manager handle the secret and renewal.

step ca certificate app.example.com app.crt app.key \
  --provisioner admin --kty ec
kubectl create secret tls app-tls --cert=app.crt --key=app.key -o yaml
kubectl apply -f app-tls.yaml
kubectl rollout restart deployment app

This path combines the strengths of episodes 9 and 10: step-ca issues quickly and cheaply for integration, cert-manager handles renewal in Kubernetes, and GitOps keeps consistency. The same pipeline can be triggered by other events — for example when deploying a new application, not only when a certificate is created.

#Closing

Episode 19 equips your PKI infrastructure with awareness and automation: expiry monitoring with openssl x509 -checkend, information-rich cert-manager status, step-ca metrics for Prometheus, disciplined alerting, then full automation through Terraform, GitOps with ArgoCD or Flux, and an idempotent issue and deploy pipeline.

Key takeaways:

• Monitor certificate validity regularly with a reasonable tolerance, not by waiting for a TLS error.
• Read the Certificate and CertificateRequest conditions in cert-manager for fast diagnosis.
• Expose step-ca metrics and scrape them with Prometheus; create alerts only for things needing action.
• Make Terraform and Helm code for issuers and certificates, complete with secure state.
• Use GitOps so PKI changes always go through review and can be rolled back.
• Design an idempotent issue and deploy pipeline so it can run repeatedly without side effects.

Your PKI is now visible and automated. But the cryptography world keeps moving. In episode 20 we look ahead: post-quantum cryptography and modern cryptography trends. See you there!