Learn Kerberos - Monitoring & Logging

#Introduction

In episode 21 you closed the gates: AES-only encryption, mandatory preauth, short lifetimes, secured keytabs, and mapped common attacks. But all that defense is blind without sight. Attacks don't knock on the door — they come in through anomalies that are only visible if you monitor logs and metrics.

Episode 22 builds the realm's "eyes": logging on the KDC and client sides, log analysis to read failure signals, monitoring tools like ELK and Grafana, and the metrics that must always be watched. This is where the detection mentioned in episode 21 becomes operational.

#Logging on the KDC Side

The MIT Kerberos KDC writes all activity to fully configurable logs. All messages — authentication, authorization, and errors — gather in one place so they can be analyzed.

#Log Location and Configuration

Logging is configured in the [logging] section of /etc/krb5.conf (or kdc.conf):

/etc/krb5.conf - KDC logging configuration

[logging]
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log
    default = FILE:/var/log/krb5libs.log

• kdc — the KDC's main activity log: all AS_REQ, TGS_REQ, and their results.
• admin_server — the log of changes made via kadmin (addprinc, modprinc, delprinc).
• default — the Kerberos library log on the machine side (client and service).

Besides FILE, the destination can be SYSLOG:severity:facility to forward to syslog (e.g. SYSLOG:INFO:LOCAL0), or DEVICE=/dev/null to discard output. Using syslog makes later SIEM integration easier.

#Log Levels

The KDC log doesn't have per-module DEBUG/INFO levels selectable like a web application. The detail level is controlled via:

• Syslog severity (INFO, ERR, NOTICE) when using SYSLOG:.
• The message classes classified by the KDC: INFO for successes, ERR for failures.
• Deep debugging when a full trace is needed, e.g. raising logging on the client side or using KRB5_TRACE (covered in the client section).

#Authentication, Authorization, and Errors

The KDC log holds three main categories:

• Authentication logs — every ticket request: who requested a TGT, from which IP, which enctypes were used, and whether it succeeded.
• Authorization logs — service ticket requests (TGS_REQ): which principal asked for access to which service.
• Error logs — failures: failed preauth, principal not found, clock skew, encryption mismatch.

The following snippet shows the message patterns you'll often encounter (the format can vary between versions; focus on the content):

AS_REQ alice@EXAMPLE.COM from 10.0.0.50: PREAUTH_FAILED
TGS_REQ alice@EXAMPLE.COM: S_PRINCIPAL_UNKNOWN
AS_REQ 10.0.0.51: CLOCK_SKEW: time is off by 360 seconds
ISSUE: authtime ... alice@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM

#Logging on the Client Side

The KDC isn't the only log source. The client side often holds answers not visible in server logs — for example when a request never reaches the KDC.

#KRB5_TRACE for Debugging

The KRB5_TRACE environment variable makes the client library write a complete trace of every Kerberos step to the specified file. Use KRB5_TRACE=/dev/stdout to watch directly:

export KRB5_TRACE=/dev/stdout
kinit alice@EXAMPLE.COM

This trace shows the whole flow: realm lookup, KDC discovery, the AS_REQ submission, the enctypes offered, up to the final result. It's the number one tool when authentication fails for an unclear reason.

#Client-Side Debugging and Application Logs

• Library logs flow to the default destination in [logging] (e.g. krb5libs.log).
• GSSAPI-based applications write their own logs; look for gssapi or kerberos in the application's logging configuration (e.g. Apache module, SSH, NFS).
• SSSD stores complete logs in /var/log/sssd/ — when SSSD fails to fetch a ticket, the trace is there, not on the KDC.

#Log Analysis

Good KDC logs will tell the story of a problem before the user gets a chance to report it. Recognize the following patterns:

• Failed authentication attempts: a surge of PREAUTH_FAILED from one IP or one user is a brute-force or password-spray signal.
• Clock skew errors: recurring appearances mean the time infrastructure has a problem — the KDC rejects tickets from machines with drifting clocks.
• Encryption type mismatches: repeated failures with encryption hints indicate an old client or a downgrade configuration.
• Principal not found errors: often caused by a wrong realm, wrong spelling, or an unregistered SPN.
• Ticket expiration issues: tickets expiring mid-way through a long job are usually because the lifetime is too short or renewal isn't set.

#Monitoring Tools

Logs alone aren't enough when they're spread across hundreds of hosts. Combine them with aggregation and visualization tools:

• Splunk / ELK for log aggregation: all KDC and client logs are forwarded to one place — via rsyslog or syslog-ng on the KDC side toward Elasticsearch/Loki, or directly to Splunk. This is where the anomaly detection from episode 21 runs as rules.
• Grafana for metrics: real-time metric visualization. Prometheus can scrape host metrics (CPU, memory, load) and custom metrics parsed from KDC logs.
• Custom monitoring scripts: simple scripts that extract the number of successful/failed AS_REQs per minute from KDC logs and export them to Prometheus.
• SIEM integration: every security-flavored alert — ticket anomalies, encryption downgrades, delegation changes — is forwarded to the SIEM for correlation with other data.

#Metrics to Monitor

Metrics are the realm's heartbeat. Start with the highest-impact ones:

• Authentication success/failure rates are the primary security metric: the normal failure-to-success distribution should be stable; a sudden change triggers an investigation.
• KDC performance measures how fast the KDC answers; start by adding a KDC replica when latency worsens (topic of episode 24).
• Database load monitors principal database health; abnormal growth signals rogue accounts.
• Ticket issuance rate reflects authentication volume; this value becomes the baseline for detecting mass ticket request attacks.
• Service response times show the end-user experience: tickets arrive on time or not.

tail -f /var/log/krb5kdc.log | grep -c "ISSUE" | while read n; do echo "asreq_ok $n"; done

Scripts like this are easily wired into a Prometheus textfile collector and visualized in Grafana — from there, alerting is just one step away.

#Conclusion

Episode 22 gave you sight: KDC logging configuration in [logging], client traces with KRB5_TRACE, analysis of failure patterns like PREAUTH_FAILED, CLOCK_SKEW, and S_PRINCIPAL_UNKNOWN, and monitoring tools (ELK, Grafana, SIEM) with the key metrics that must be monitored.

Key takeaways:

• Point the KDC, admin server, and library logs to files or syslog via [logging] in krb5.conf.
• KRB5_TRACE=/dev/stdout reveals the entire authentication flow from the client side.
• Recognize the log patterns: PREAUTH_FAILED for passwords, CLOCK_SKEW for time, S_PRINCIPAL_UNKNOWN for principals.
• Monitor the key metrics: success/failure ratios, KDC performance, and ticket issuance rate — and alert on them.

In episode 23 you'll use all these eyes to heal: Kerberos Troubleshooting — mapping common errors to their solutions, debugging techniques with KRB5_TRACE and klist -e, and network and keytab problems. See you there!