Learn Kerberos - High Availability

#Introduction

In episode 24 you tuned KDC and client performance. Episode 25 covers the natural continuation of that theme: high availability. Performance is useless if the service isn't available. Here you arrange a KDC architecture that keeps serving authentication even when one component dies — from KDC replicas, database propagation, automatic failover via DNS SRV, to split-brain prevention and disaster recovery.

#The Roles of Master and Replicas

In the standard MIT Kerberos model there is one master KDC that is the only place for writing data, and several replicas serving read requests. All changes — adding principals, changing passwords, modifying policies — are done on the master, then propagated to the replicas.

This distinction matters: a replica isn't an "equivalent second KDC" — a replica is a reader of the master's database. Clients don't distinguish between the two, and that's exactly what you want.

#Database Propagation with kprop

Changes on the master don't automatically reach replicas; you must send a copy of the database periodically. The first step is producing a database dump on the master:

kdb5_util dump /var/lib/krb5kdc/dump/replica_datatrans

On the replica side, the kpropd daemon listens on TCP port 754 and accepts incoming databases. The list of principals allowed to send is written in kpropd.acl:

kpropd.acl on the replica

host/kdc-master.example.com@EXAMPLE.COM
host/replica1.example.com@EXAMPLE.COM

Each line is a host/ principal allowed to propagate; after editing this file, restart kpropd. By default kpropd immediately runs the database load after receiving data, so make sure the file sent is always a valid dump — a corrupted transfer can overwrite the replica database.

From the master, send the dump to the replica:

kprop -f /var/lib/krb5kdc/dump/replica_datatrans replica1.example.com

kprop sends the dump file encrypted using the host/ principal. To schedule propagation, put this command in cron or a systemd timer — e.g. every hour or every few minutes, depending on how quickly you want changes reflected.

#Incremental Propagation with iprop

Full propagation (a full dump) costs in proportion to the database size. For frequent changes, MIT provides iprop — incremental propagation that only sends the changes:

Enabling iprop in krb5kdc.conf

[realms]
    EXAMPLE.COM = {
        iprop_enable = true
        iprop_logfile = /var/lib/krb5kdc/iprop.log
        iprop_slave_poll = 60
    }

With iprop, replicas poll the master every iprop_slave_poll seconds and pull the changes since the last log entry. Many deployments use a combination: iprop for routine synchronization, full dumps as a periodic backup.

#Automatic Failover

When the primary KDC can't be reached, clients must know there's another KDC. Instead of writing a static KDC list, let clients discover the KDC via DNS. MIT supports SRV lookup for the following names:

• _kerberos._udp.EXAMPLE.COM and _kerberos._tcp.EXAMPLE.COM — the KDCs for UDP and TCP.
• _kerberos-master._tcp.EXAMPLE.COM — points to the master, used for kadmin.
• _kpasswd._udp.EXAMPLE.COM and _kpasswd._tcp.EXAMPLE.COM — for the password change server.

Example in the DNS zone:

SRV records in the DNS zone

_kerberos._udp.EXAMPLE.COM.   IN SRV 0 100 88  kdc1.example.com.
_kerberos._tcp.EXAMPLE.COM.   IN SRV 0 100 88  kdc1.example.com.
_kerberos._udp.EXAMPLE.COM.   IN SRV 1 100 88  kdc2.example.com.
_kerberos._tcp.EXAMPLE.COM.   IN SRV 1 100 88  kdc2.example.com.
_kerberos-master._tcp.EXAMPLE.COM. IN SRV 0 100 88 kdc1.example.com.
_kpasswd._udp.EXAMPLE.COM.    IN SRV 0 100 88  kdc1.example.com.

Priority 0 for kdc1 and 1 for kdc2 makes clients try kdc1 first, then switch when it fails. On the client side, enable this lookup with dns_lookup_kdc = true in krb5.conf.

#Client Failover Behavior

The Kerberos library on the client side tries KDCs one by one in SRV order; when the first KDC doesn't answer within kdc_timeout, the client moves to the next. Failover feels automatic — provided the timeout and retry values are set sensibly (see episode 24). Remember: SRV records aren't a health check — clients still try a recorded KDC even if it's been dead for a long time; what saves you is the priority order and the switch on timeout.

#Health Checking and Load Balancing

Because SRV isn't a health monitor, you need your own monitoring: check port 88 (TCP and UDP) from several network points, run a test kinit to each KDC from a monitoring host, and watch KDC logs for failure spikes. For load balancing, use priority and weight in SRV or a load balancer in front of the KDC pool — make sure write traffic and kadmin still only go to the master.

#Split-Brain Prevention

The biggest danger of a multi-KDC architecture is split-brain: two "masters" both believing they're writers and changing the database at the same time, producing contradictory data. The MIT model avoids this with a simple rule: there is only one master, and the whole system is arranged around that rule.

Replicas are read-only: all write requests are rejected at a replica, and an application trying kadmin to a replica will fail. This is the partition wall maintaining consistency — not because the replica can't write, but because policy and configuration enforce it.

Consistency is maintained by the single propagation direction (master to replica) and the synchronization schedule; the consequence is a lag between a change on the master and its appearance on the replica. Understand and measure your propagation delay (eventual consistency is a feature, not a bug), don't put data needing real-time consistency on the read-only side alone, and periodically verify that the replica's database contents align with the master.

#Disaster Recovery

The KDC is the realm's center of trust; losing the principal database means the realm must be rebuilt from scratch. A KDC backup at minimum includes: the principal database, the master key file (/var/lib/krb5kdc/.k5.EXAMPLE.COM or stash), the configuration files (krb5.conf, krb5kdc.conf, kadm5.acl, kpropd.acl), and the iprop log so replaying changes can continue.

#Database Dump and Restore

kdb5_util dump produces a text file containing all principals — portable and reloadable:

kdb5_util dump /backup/kerberos/principal.dump
kdb5_util load /backup/kerberos/principal.dump

kdb5_util is the key pair for DR: dump to save, load to restore. Keep the dump in a protected location and, ideally, at a different site.

#Backing Up the Master Key

The master key is the key that encrypts the entire database; without the master key, a dump is unusable. The master key is stored as a stash for automatic KDC use — copy the stash file to a safe place and to backup media. The critical point: the master key must be identical across all replicas. When building a replica, you copy the stash and database from the master, not create a new key. Losing the master key at every location means the KDC data can never be read again.

#Testing Recovery

A backup that has never been tested isn't a backup. A regular recovery drill includes: building a new KDC from a dump on a test machine, restoring from the master key backup, making sure clients can log in after the restore, and measuring the restore time against the agreed RTO (Recovery Time Objective). This testing also trains the team — when the real event happens, the procedure has already been run.

#Multi-site Deployments

When a realm stretches across several locations, putting all KDCs in one site makes remote clients depend on slow, fragile WAN links. Every login needs several round-trips to the KDC, so clients that must ask another site bear WAN latency for every authentication step.

The common model: one master in the primary site, one or more replicas in each site. Clients in each site are directed to their local replica via tailored DNS SRV (e.g. split-horizon DNS or geographic resolution), so inter-site traffic only appears during propagation, not during every login.

Inter-site propagation uses the same mechanisms (kprop or iprop) with extra considerations: schedule full propagation during quiet hours, consider iprop to reduce transfer volume, monitor replication lag per site, and make sure the inter-KDC network is protected from interference — the encryption and authentication of propagation are already handled by Kerberos via the host/ principal.

#Conclusion

Episode 25 laid the reliability foundation: the roles of master and replicas, database propagation with kprop and iprop, kpropd.acl configuration, automatic failover via DNS SRV, split-brain prevention with the single-master model, disaster recovery procedures, and multi-site KDC deployments.

Key takeaways:

• One master, many replicas — the model that prevents split-brain and provides read capacity.
• Propagation isn't instant synchronization — understand kprop/iprop and their schedules; replicas always have a little lag.
• DNS SRV for failover — clients switch KDCs automatically, but only if timeouts are set sensibly.
• The master key is everything — store it, secure it, and test its restore.

In the next episode, episode 26, you scale even further: large-scale deployments — hierarchical realms, regional KDCs, central vs distributed models, anycast for discovery, and automation with Ansible and Infrastructure as Code.