Learn Kerberos - Troubleshooting

#Introduction

In episode 22 you built the realm's eyes: configured KDC logs, client traces with KRB5_TRACE, failure pattern analysis, and monitored metrics. Now it's time to use all of that to heal. Kerberos errors can look confusing, but almost all of them come from six same causes.

Episode 23 is a field troubleshooting guide: mapping common errors to their causes and solutions, debugging techniques from the client side to packet capture, network and time diagnosis, and the keytab problems that most often plague administration.

#Common Problems and Their Solutions

Most Kerberos errors fall into one of the following six categories. Memorize this table — it will be your first reference:

#Clock Skew Too Great

The KDC rejects requests if the client-KDC clock difference exceeds the tolerance — the MIT default is 5 minutes. The most common symptom is Clock skew too great on kinit. Verify the time with timedatectl status and check whether NTP is running.

#Server Not Found in Kerberos Database

The KDC doesn't recognize the requested principal. This is usually because the SPN isn't registered, the realm is wrong, or the account doesn't exist in AD. Check with kadmin getprinc and compare the realm spelling with the configuration.

#Preauthentication Failed

The KDC rejects the preauth proof — most often because the password is wrong, but it can also be a drifting clock causing the preauth timestamp to be rejected. Try a manual kinit with a trace to see the details.

#Ticket Expired

A TGT is only valid for its lifetime. Long jobs that exceed the validity period fail with an expired ticket; set a renewlife or re-run kinit.

#Cannot Contact KDC

The client can't reach the KDC — a network or DNS problem. Check ports 88/tcp and 88/udp toward the KDC, and the realm's name resolution.

#Credential Cache Not Found

The library can't find the ticket cache pointed to by KRB5CCNAME. The cache can move when switching users or sessions; set the variable to the correct path or re-run kinit.

#Debugging Techniques

When the error is unclear, step back and use debugging tools — don't guess.

#KRB5_TRACE for a Complete Trace

The client-side trace shows every step the library performs — the chosen realm, the contacted KDC, the offered enctypes, and the raw error message:

export KRB5_TRACE=/dev/stdout
kinit alice@EXAMPLE.COM

Look for lines containing error or failed in the trace output; the KDC message usually names the real cause — e.g. KRB5KDC_ERR_PREAUTH_FAILED or KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN.

#klist -e for Encryption

Check the enctypes of the stored tickets. An encryption mismatch between client, KDC, and service is the source of the KDC has no support for encryption type error:

klist -e

If a ticket shows up with the RC4 enctype when AES should be used, there's a downgrade configuration — go back to the hardening in episode 21.

#kadmin getprinc for Principal Details

Check principal details directly from the KDC database — validity, flags, and key KVNO:

sudo kadmin.local -q "getprinc alice@EXAMPLE.COM"

#Wireshark and tcpdump

When the problem is in the network or protocol, capture Kerberos traffic. Use tcpdump to capture on the server side, Wireshark for analysis:

sudo tcpdump -i eth0 -n port 88 -w kerberos.pcap

In Wireshark, the krb5 filter shows AS_REQ, AS_REP, TGS_REQ, and TGS_REP. Compare failed packets with successful ones to find the difference — often the answer is in the packet's error-code.

#Network Debugging

Kerberos depends entirely on DNS and connectivity. Check both before blaming the KDC.

#DNS Resolution Checks

The client finds the KDC via the realm's SRV records. Make sure the records exist and can be resolved:

dig +short _kerberos._tcp.example.com SRV
dig +short dc01.example.com A
getent ahosts dc01.example.com

#Port Connectivity

The KDC serves on ports 88/tcp and 88/udp (AS/TGS), and 749/tcp for kadmin. Test the connections:

nc -zv dc01.example.com 88
nc -uzv dc01.example.com 88
nc -zv dc01.example.com 749

#Firewall Rules

Firewall-blocked ports cause Cannot contact any KDC. Check the rules on the client and server, and make sure 88/tcp, 88/udp, and 749/tcp are open from the right networks.

#Packet Capture Analysis

If the TCP connection succeeds but the request still fails, capture on both sides. A packet sent with no reply means the packet is blocked; a reply with an error code means a protocol problem.

#Time Synchronization Problems

Time is Kerberos's silent enemy. Poor time synchronization produces misleading errors — Clock skew too great or even Preauthentication failed.

• Clock skew detection: timedatectl shows the synchronization status; compare with the KDC time using timedatectl or chronyc.
• NTP verification: make sure chronyd or ntpd is running and locked to a time source:

timedatectl status
chronyc sources -v
chronyc tracking

• Time source issues: the KDC and all clients must use the same time source — preferably the KDC itself or a shared trusted source, not different ones.
• Timezone problems: a wrong zoneinfo makes the clock look right on screen but still off in epoch terms. Set TZ correctly and keep the clock stored in UTC.

#Keytab Troubleshooting

A faulty keytab causes machine and service authentication to fail sporadically. Check with klist:

klist -kt /etc/krb5.keytab

• KVNO mismatches: each keytab entry has a key version (KVNO). If the account key in AD was reset (KVNO went up) but the keytab still holds the old KVNO, authentication fails with KRB_AP_ERR_BADKEYVER. Compare the keytab KVNO with the value from kadmin getprinc; regenerate the keytab if they differ.
• Missing entries: the keytab doesn't hold the needed SPN. Compare the klist -kt output with the account's SPN list; add the entry via msktutil add-sph or ktpass.
• Permission issues: a keytab readable by the application process — or, on the contrary, unreadable because of wrong permissions — surfaces errors when a service tries to authenticate. Make sure the file is owned by the service user with an appropriate mode.
• Expired keys: a password reset in AD makes the keytab keys expire. Recreate the keytab after resetting a service account password.

msktutil add-sph --service HTTP/web01.example.com \
  --keytab /etc/krb5.keytab

#Conclusion

Episode 23 closed the troubleshooting loop: six common errors mapped to their causes and fixes, debugging techniques from KRB5_TRACE to tcpdump explained, network and time diagnosed step by step, and keytab problems like KVNO mismatches and permissions solved.

Key takeaways:

• Six main errors — clock skew, principal not found, preauth, expired, cannot contact KDC, and cache not found — cover almost every problem.
• KRB5_TRACE=/dev/stdout, klist -e, and kadmin getprinc are the main debugging trio.
• Work the network in order: DNS, port 88, then packet capture.
• Time and keytabs are the two most common problem sources — verify both first.

In episode 24, your question changes from "why does it fail" to "why is it slow": Performance Tuning — optimizing the KDC database, KDC replicas for load balancing, worker threads, and caching strategies. See you there!