Learn Kerberos - Large-Scale Deployments

#Introduction

In episode 25 you built high availability with KDC replicas, kprop propagation, DNS SRV failover, and disaster recovery. Episode 26 widens the scale to the enterprise level: large-scale deployments. When a realm must serve tens of thousands of users in many locations, a single-realm architecture with a few replicas starts to show its limits. You'll see hierarchical realm design, the central vs distributed model, scalability patterns like read replicas and anycast, and how automation manages operational complexity.

#Enterprise Architecture

#Hierarchical Realm Design

As an organization grows — subsidiaries, divisions, countries — a single realm often becomes too large for one team to manage. The classic solution is a hierarchical realm: one root realm (e.g. EXAMPLE.COM) with child realms (e.g. ASIA.EXAMPLE.COM, EU.EXAMPLE.COM). Trust between child realms and the root forms automatically, and capaths enable transitive trust between realms. The advantages: each realm is managed by its own team, policies can differ, and one realm's failure doesn't roll over the whole organization.

#Regional KDCs

In a hierarchical architecture, each region has its own KDC — a local master plus local replicas. Clients in Asia ask the ASIA KDC, clients in Europe the EU KDC. Inter-realm traffic only happens when a user genuinely needs a resource in another realm.

#Central vs Distributed Model

Two design poles, with many variations in between:

The choice isn't right-or-wrong: many organizations start central and evolve to distributed as they grow — a gradual transition that leverages the multi-site replica concept from episode 25.

#Scalability Patterns

#Read Replicas

From episode 25 you already know replicas as database readers. At large scale their role grows more important: most traffic is read requests (AS and TGS), and each replica adds horizontal capacity without touching the master. The rule remains: master writes, replicas read.

#Anycast for KDC Discovery

Anycast advertises one identical IP address from many locations; routers direct each client to the nearest instance. For KDC discovery, this makes clients always land on the nearest KDC without complicated per-client configuration. Several KDCs (or load balancers in front of them) advertise the same anycast address; routing (BGP or anycast at the DNS level) carries packets to the nearest instance. If one site dies, routing automatically moves to another site — failover at the network level, before clients notice anything changed.

#CDN-Style Distribution

The CDN pattern — placing "copies" of a service near users — translates in Kerberos to placing a KDC replica at every site, exactly like an edge cache server in a CDN. Clients are always served by the nearest instance; data stays consistent through propagation from the master. It's the combination of anycast and per-site replicas that makes the authentication experience uniform wherever the user is.

#Performance at Scale

#Database Sharding

At very large scale, a single KDC database can become the limiting point. Sharding — splitting the database into several parts — is conceptually possible by separating data per realm or per principal subset. But let's be honest: sharding a KDC database is hard to apply to MIT Kerberos because the database functions as one unit per realm. A realistic approach: separate load between realms (child realms have their own database — natural sharding via hierarchy), routinely clean up unused principals, and scale up capacity with adequate memory and disk.

#Caching Layers

At large scale, caching becomes architecture, not a trick:

• Client side — a good ccache avoids most requests to the KDC.
• Resolver side — DNS caching for KDC SRV records dampens lookup spikes.
• Application side — GSSAPI applications keep service tickets while valid, lowering per-request traffic to the KDC.

The more requests answered by cache at lower layers, the less load reaches the KDC.

#Connection Pooling and Horizontal Scaling

Applications that repeatedly use Kerberos authentication must pool connections and contexts instead of creating them from scratch every time — this reduces expensive TCP handshakes and GSSAPI negotiations. For horizontal scaling, adding replicas is the standard action; the master itself is harder to scale horizontally because of its single-writer role, and that's where a multi-realm architecture provides a more natural scaling path.

#Operational Challenges

#Managing Large Numbers of Principals

With tens of thousands of principals, one-by-one operations via kadmin no longer make sense. Write scripts that use kadmin.local with batch input, use a file full of kadmin commands processed at once, and enforce naming discipline for principals and metadata so they're easy to identify and clean up.

#Keytab Distribution

Every service needs a keytab; at large scale its distribution must be encrypted and audited: distribute via a secure, authenticated channel (not email or public storage), record who holds which keytab and for what, and rotate service keytabs periodically and remove unused ones. kadmin.local adds a keytab with ktadd and removes it with ktremove; this process can be automated via configuration management.

#Password Rotation

Password rotation for service principals is done by changing the key (change key), which automatically raises the kvno. Use ktadd to refresh the service key, synchronize the rotation timing so applications and their keytabs update together (replacing the key without updating the keytab makes the service lose access), and make periodic rotation a natural audit — principals not rotated are usually no longer in use.

#Monitoring Complexity

More KDCs means more logs and metrics; the complexity grows faster than the component count. Aggregate metrics in one place (Grafana and Prometheus — the theme of episode 22), monitor meaningful metrics (authentication success rates, latency, propagation lag, database size), and apply centralized alerting so failure spikes are visible before they affect users.

#Automation

#Infrastructure as Code

At large scale, a KDC must not be built manually. Infrastructure as Code describes the entire deployment — servers, configuration, network — as versionable, testable code. When a replica breaks, replacing it is a re-run, not guesswork.

#Configuration Management with Ansible

Ansible (or Puppet) manages KDCs and clients centrally. Configuration templates ensure all nodes are consistent:

krb5.conf template for all nodes

[libdefaults]
    default_realm = ASIA.EXAMPLE.COM
    dns_lookup_kdc = true
    rdns = false
 
[realms]
    ASIA.EXAMPLE.COM = {
        kdc = anycast-kdc.example.net
        admin_server = kdc-master.example.com
    }

The same template rendered with different values per role (master, replica, client) keeps configuration drift — the main cause of Kerberos problems in production — under control.

A brief playbook example for the replica role:

- name: Provision replica KDC
  hosts: kdc-replicas
  tasks:
    - name: Install packages
      ansible.builtin.package:
        name: krb5-kdc
        state: present
    - name: Deploy kpropd.acl
      ansible.builtin.copy:
        dest: /etc/krb5kdc/kpropd.acl
        content: "host/kdc-master.example.com@EXAMPLE.COM\n"

The kpropd.acl in this example is still simple — load the host/ principal of every master and replica allowed to propagate.

#Automating Principal Provisioning

Provision principals automatically when a new server is born, not afterwards by hand:

kadmin.local -q "addprinc -randkey host/new-server.example.com"
kadmin.local -q "ktadd -k /etc/keytabs/host.keytab host/new-server.example.com"

kadmin with -randkey creates a principal without a password; ktadd then exports the key to a keytab at the desired location. Fold these steps into the provisioning playbook so every new server is immediately authenticated correctly.

#Self-Service Portal

At large scale, helpdesk can't handle all manual requests. Self-service moves the load: self password resets via kpasswd or a web portal connected to kadmin, service principal requests through an auto-approved workflow (ticket, review, then script execution), and keytab rotation triggered by the service owner. Every self-service action must have its logs audited — automation reduces burden, not accountability.

#Conclusion

Episode 26 closed the discussion of running Kerberos at enterprise scale: hierarchical realms and regional KDCs, the central vs distributed model, read replicas, anycast for discovery, CDN-style distribution, database sharding and caching, the challenges of principal, keytab, password, and monitoring management, and automation with IaC, Ansible, and self-service portals.

Key takeaways:

• Realm hierarchy is the natural scaling path — each realm has its own master, each region is served by a local KDC.
• Most traffic is reads — replicas, anycast, and caching handle the load long before the master is touched.
• Consistency needs discipline — at large scale, configuration drift and propagation lag are the main enemies.
• Automation is a force multiplier — IaC and self-service are the only way to manage complexity that grows.

In the next episode, episode 27, the series moves to a new location: Kerberos in the Cloud — deployments on AWS, GCP, and Azure, hybrid environments, and running KDCs in containers and Kubernetes.