Learn Apache Kafka - Authentication with SASL

#Introduction

A PLAINTEXT listener lets anyone who can reach the broker port connect. In production, you must know who is connecting before deciding what they're allowed to do — this is the job of authentication. Kafka uses the SASL (Simple Authentication and Security Layer) framework for this mechanism.

Episode 16 covers the four SASL mechanisms Kafka supports: PLAIN, SCRAM, GSSAPI (Kerberos), and OAUTHBEARER. You'll learn broker-side and client-side configuration, the structure of JAAS files, creating SCRAM users with kafka-configs.sh, and OAuth 2.0 integration with an identity provider.

#SASL Mechanisms

#SASL/PLAIN

PLAIN sends username and password in plain text — so it must be used together with TLS encryption (SASL_SSL). Simple and suitable for development or integration with systems that already store static passwords. There's no hash support on the broker; credentials must be handled by another mechanism like LDAP or a JAAS file.

#SASL/SCRAM

SCRAM (Salted Challenge Response Authentication Mechanism) is a challenge-response mechanism that never sends a password over the network. Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. Credentials are stored on the broker as salted hashes and created with kafka-configs.sh — this is the most recommended mechanism for modern deployments.

#SASL/GSSAPI and SASL/OAUTHBEARER

• GSSAPI: Kerberos authentication common in large enterprises; requires a KDC and a Kerberos principal per service.
• OAUTHBEARER: OAuth 2.0 authentication with access tokens; credentials are centralized in an identity provider and support short-lived tokens. Confluent and many cloud setups use this mechanism.

#SASL Configuration

#Broker Side

For SCRAM, enable the mechanisms in server.properties:

listeners=SASL_SSL://:9093
listener.security.protocol.map=SASL_SSL:SASL_SSL
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512

listeners=SASL_SSL://:9093 means all connections go through TLS and SASL. sasl.enabled.mechanisms registers the mechanisms that may be used, and sasl.mechanism.inter.broker.protocol determines the mechanism for inter-broker communication.

#JAAS Files

Broker and client credentials are provided through JAAS (Java Authentication and Authorization Service) files:

KafkaServer {
  org.apache.kafka.common.security.scram.ScramLoginModule
    required username="kafka-broker" password="<password>";
};

For SCRAM, JAAS is only used at startup; the real credentials are fetched from cluster metadata. Don't commit JAAS files to the repository — store them as secrets and restrict file permissions.

#Implementing SCRAM

#Creating Users with kafka-configs

SCRAM users are created directly in cluster metadata:

bin/kafka-configs.sh --bootstrap-server localhost:9093 \
  --alter --add-config 'SCRAM-SHA-512=[password=strong-pass]' \
  --entity-type users --entity-name app-producer

kafka-configs.sh --alter --entity-type users creates credentials for the app-producer principal. The password is stored as a salted hash in metadata; after this, app-producer can use that user for authentication.

#Client Configuration

Clients provide username and password through sasl.jaas.config:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule \
  required username="app-producer" password="strong-pass";

security.protocol=SASL_SSL requests encryption and authentication, and sasl.jaas.config carries the client credentials. Note: clients also need a truststore to verify the broker certificate (TLS details in episode 18).

#Credential Management

Don't put passwords in version-controlled configuration. Best practices: fetch them from a secret manager (Vault, AWS Secrets Manager) when the application starts, or use variable substitution. Rotate passwords periodically with --alter --add-config without stopping clients.

#OAuth Integration

#OAuth 2.0 Authentication

With OAUTHBEARER, clients obtain an access token from the identity provider and send it to the broker. The broker validates the token — usually via an introspection endpoint or JWT signature validation — without storing user credentials:

security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.login.callback.handler.class=com.example.OAuthLoginCallbackHandler
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule \
  required clientId="app" clientSecret="secret" scope="kafka";

sasl.mechanism=OAUTHBEARER enables the token flow. The login handler manages token acquisition, while the broker uses sasl.server.callback.handler.class for validation.

#Token Validation and Identity Provider

Brokers validate tokens through introspection to the identity provider (for example Keycloak, Okta, Auth0) or by verifying the JWT signature with a public key. Security policies, such as audience and expiry, are determined by the issuer. The main advantage: user management, credential rotation, and revocation are centralized in the identity provider.

#Custom OAuth Handlers

For special needs — for example an internal token format or introspection caching — you implement AuthenticateCallbackHandler and LoginCallbackHandler. This gives full flexibility to integrate Kafka with an enterprise auth ecosystem.

#Closing

In this episode 16 you've understood the four SASL mechanisms, broker and client configuration with JAAS files, creating SCRAM users with kafka-configs.sh, and OAuth 2.0 integration with identity providers and custom handlers.

The key takeaways:

• SASL_SSL combines authentication and encryption in a single connection.
• SCRAM-SHA-512 is the recommended mechanism for new deployments.
• SCRAM users are created with kafka-configs.sh and stored as hashes in metadata.
• JAAS provides credentials; never commit it to a repository.
• OAUTHBEARER moves user management to the identity provider.
• Validate OAuth tokens via introspection or JWT signature.

In the next episode 17 we'll determine what authenticated users are allowed to do: authorization with ACLs. You'll learn principals, resource types, operations, creating ACLs with kafka-acls.sh, least privilege patterns, and custom authorizers.