Learn PKI - High Availability & Multi-CA

#Introduction

Episode 17 dissected advanced X.509 formats: extensions, SAN, and the encoding differences between DER and PEM. You now understand that a certificate is only a document — its strength comes from who signs it and how the surrounding infrastructure protects the chain of trust. This is the point where many teams hit the next wall: how do you scale a CA originally built for one server to serve an entire organization without becoming a single point of failure?

Episode 18 answers that question. We will build high availability and multi-CA: separating intermediate CAs per environment, cross-signing to preserve compatibility during root migration, low-risk root rotation, HA strategies for step-ca and cert-manager, and close with recovery scenarios. By the end of this episode, you will have a roadmap for raising PKI from homelab scale to production scale.

#Why a Single CA Is Not Enough

A single CA feels simple: one root, one intermediate, one key to manage everything. But in a real environment, that simplicity turns into risk. One CA shared for development, staging, and production means a single configuration error in a test environment can poison the trust chain of the production environment. A single CA also means no separation of duties: an admin who can issue development certificates automatically can also issue production certificates.

The principle from episode 16 is relevant here again: separation of authority. The smaller the scope of a CA, the easier it is to secure, audit, and limit when an incident occurs. Multi-CA is not about adding complexity without reason — it is a structural way to keep one failure from taking everything down.

#Multiple Intermediate CAs per Environment

A model commonly used by many organizations is one offline root CA for the whole company, then one set of intermediate CAs for each environment. The root acts as the highest trust source that is rarely accessed, while each intermediate issues certificates in its own environment.

root-ca:
  type: offline
  lifetime: 20y
  intermediates:
    - name: intermediate-dev
      profile: short-lived
    - name: intermediate-staging
      profile: standard
    - name: intermediate-prod
      profile: production

This structure gives several advantages at once. First, revocation becomes more targeted: if the dev intermediate leaks, we revoke certificates only for the dev chain without touching production. Second, policy can differ per environment — dev certificates have a short validity, production longer. Third, a compromise in one environment does not drag other environments into the same trust chain.

openssl x509 -req -in intermediate-dev.csr \
  -CA root-ca.crt -CAkey root-ca.key \
  -CAcreateserial -out intermediate-dev.crt \
  -days 730 -extfile intermediate-dev.ext

This process runs on the offline machine where the root lives, then the result is carried out for distribution. The intermediate private key must also be isolated: store it in an HSM or KMS as discussed in episode 13, and restrict access to the admins assigned to that environment.

#Cross-signing Between Roots

Cross-signing is a technique that makes two root CAs trust each other indirectly: the old root signs a certificate for the new root, and vice versa. The result: clients that trust the old root also accept certificates from the new root's chain, without needing to replace their trust store first.

openssl x509 -req -in new-root.csr \
  -CA old-root.crt -CAkey old-root.key \
  -CAcreateserial -out new-root.cross.crt \
  -extfile new-root.ext

When is cross-signing needed? The most common case is root migration: thousands of devices in the field still trust the old root and cannot be updated at once. With cross-signing, old clients can still validate certificates issued under the new root during the transition period. Check the chain result with openssl verify -verbose -CAfile old-root.pem new-chain.pem to make sure the validation path is correct before wide distribution.

#Root Rotation

Root rotation is the process of replacing the root CA with a new one — usually because the old key is nearing the end of its validity, cryptographic standards have changed, or there is suspicion of compromise. This is one of the riskiest operations in the PKI world because the entire client trust store must be updated.

A safe path for root rotation: first, issue a new root with the latest algorithm and key size. Second, cross-sign the old root with the new root so the transition period does not break services. Third, distribute the new root to all internal trust stores gradually, while adding the old root to a revoked list so no one can abuse it. Finally, when legacy traffic is zero, deactivate the old root and clean it from all systems.

#Distributed step-ca and High Availability

step-ca from episode 9 does have state, but it can still be run in a distributed fashion. The key is the choice of backend and key storage. For a multi-instance setup, put step-ca behind a load balancer with two or more instances, share the data store among the instances, and store the CA key in an HSM via PKCS#11 so the key never leaves the hardware.

step ca init --name "Production Root" --dns ca.example.com --address ":8443" \
  --provisioner admin --with-ca-url https://ca.example.com
step-ca ./config/ca.json --badger-dir ./data/ca \
  --kms-uri pkcs11:module-path=/usr/lib/softhsm/libsofthsm2.so

When one instance dies, other instances keep serving requests as long as the shared data store is reachable. Routine health checks help us know when an instance needs replacement — get into the habit of checking with step ca health regularly and including it in monitoring, which we will build in episode 19.

#cert-manager High Availability

cert-manager from episode 10 also has a high availability mode. A default deployment with one replica is enough for a lab, but for production we raise the replica count and enable leader election. With leader election, only one controller processes certificates at a time, while other pods are ready to take over when the leader dies.

replicas: 3
leaderElection:
  enabled: true
  namespace: cert-manager
extraArgs:
  - --max-concurrent-changes=30

The active controller writes status to the Certificate object; other pods only watch. If the leader fails, a follower fails over within seconds without manual intervention. Monitor events in the namespace with kubectl get events -n cert-manager to see leader changes and errors before they become big problems.

#Recovery Scenarios

HA is not about avoiding failure entirely — that is impossible. HA is about knowing what happens when a component fails. Let us discuss three common scenarios you must rehearse.

#Lost Intermediate CA

If an intermediate CA key is lost, the impact is not as severe as losing the root, but it is still significant. Old certificates stay valid until their validity ends because clients only need the certificate and chain, not the key. What is lost is the ability to issue new certificates. The solution: issue a new intermediate from the root, distribute the new chain, and update the CA bundle on all clients. All old certificates remain validatable until expiry.

#Compromised Root CA

A compromised root is the worst-case scenario. First step: immediately revoke all certificates in that chain via CRL and OCSP. Second, create a new root and run the rotation process we discussed. Third, publish incident communication and the transition timeline. Document all the steps as a playbook before an incident happens — when panicking, the brain cannot be reasoned with.

#Split Brain and Partial Outage

When half the step-ca or cert-manager instances die, the healthy system must keep serving. On the step-ca side, the remaining instances continue issuance as long as the shared data store is reachable. On the cert-manager side, leader election ensures only one controller acts; if the leader dies, a follower takes over automatically. Deliberately rehearse this scenario so the team is not shocked when it actually happens.

kubectl scale deploy cert-manager --replicas=0
kubectl scale deploy cert-manager --replicas=3
kubectl get certificate --all-namespaces

#Closing

Episode 18 takes your PKI from one server to a production-ready architecture: separating intermediate CAs per environment, cross-signing for smooth transitions, measured root rotation, high availability for step-ca and cert-manager, and recovery playbooks for worst-case scenarios.

Key takeaways:

• Separate intermediate CAs per environment so one incident does not drag the whole organization down.
• Use cross-signing during root migration so old clients can still validate the new chain.
• Design gradual root rotation: issue, cross-sign, distribute, then retire the old root.
• step-ca can run distributed with a shared data store; store the key in an HSM or KMS.
• cert-manager HA uses replicas and leader election for automatic failover.
• Rehearse recovery scenarios regularly and keep them all as a written playbook.

HA without observation is a blind journey. In episode 19 we install the eyes: monitoring, observability, and an automation pipeline so every certificate and every CA is always watched. See you there!