Learn Apache Kafka - Authorization with ACLs

#Introduction

Authentication answers the question who are you. Authorization answers the next one: what are you allowed to do? In Kafka, the answer is governed by ACLs (Access Control Lists) — rules that connect principals with operations on specific resources.

Without proper authorization, every authenticated user could write to any topic, read any data, even delete topics. In episode 17 you'll learn the concepts of principals, resources, and operations, manage ACLs with kafka-acls.sh, apply ACL patterns for producers and consumers, and understand least privilege and custom authorizers.

#ACL Concepts

#Principals and Resource Types

A principal is the identity produced by authentication, usually User:app-producer with SASL/SCRAM or User:CN=client with SSL. ACLs allow or deny a principal operating on a resource. The four main resource types:

• Topic: the data being written and read.
• Group: consumer groups; controls who may consume.
• Cluster: cluster-level operations like topic creation and reassignment.
• TransactionalId: transaction operations; controls who may use a transaction ID.

#Operations

Each resource has specific operations:

• Topic: Read, Write, Describe, Create, Delete, Alter.
• Group: Read (allows joining and committing offsets), Describe.
• Cluster: Create, Alter, Describe, ClusterAction, IdempotentWrite.

#How Authorization Works

When a broker receives a request, the authorizer checks: is there an ACL allowing this principal to perform this operation on this resource? If not, the request is denied. Note: default deny — without ACLs, everything is denied when allow.everyone.if.no.acl.found=false (the default).

#Managing ACLs

#Creating ACLs

ACLs are created with kafka-acls.sh:

bin/kafka-acls.sh --bootstrap-server localhost:9093 \
  --add --allow-principal User:app-producer \
  --operation Write,Describe \
  --topic orders

The kafka-acls.sh --add command adds an ACL. The app-producer principal can now write to and describe the orders topic. Note that the Describe operation often needs to be added explicitly because clients need it for metadata.

#Listing and Removing ACLs

bin/kafka-acls.sh --bootstrap-server localhost:9093 --list --topic orders
bin/kafka-acls.sh --bootstrap-server localhost:9093 \
  --remove --allow-principal User:app-producer \
  --operation Write --topic orders

kafka-acls.sh --list shows ACLs for a resource, and kafka-acls.sh --remove removes rules that are no longer needed. Regularly auditing unused ACLs is part of long-term security.

#Wildcard ACLs

--resource-pattern-type with the LITERAL or PREFIXED value determines matching. Wildcards use --topic '*' or a prefixed pattern to cover many topics at once:

bin/kafka-acls.sh --bootstrap-server localhost:9093 \
  --add --allow-principal User:analytics \
  --operation Read --resource-pattern-type PREFIXED \
  --topic analytics-

--resource-pattern-type PREFIXED matches all topics starting with analytics-. Prefixed patterns are safer than '*' because their scope is clear.

#Authorization Patterns

#Producer and Consumer ACLs

The minimal common production pattern:

bin/kafka-acls.sh --bootstrap-server localhost:9093 \
  --add --allow-principal User:app-producer \
  --operation Write,Describe --topic orders
 
bin/kafka-acls.sh --bootstrap-server localhost:9093 \
  --add --allow-principal User:app-consumer \
  --operation Read,Describe --topic orders
 
bin/kafka-acls.sh --bootstrap-server localhost:9093 \
  --add --allow-principal User:app-consumer \
  --operation Read --group order-consumers

Consumers need Read on the topic and Read on the group they use — both are mandatory. Without the group ACL, consumers fail to join the group even if the topic ACL is correct.

#Admin ACLs and Least Privilege

Admin applications use ClusterAction and Alter on the cluster — these principals must be tightly restricted. Apply the least privilege principle: every principal gets only the minimum operations needed. Use different service accounts for producers and consumers, not a single super user account.

#Service Account Management

Create a separate service account per application instead of using human accounts. Rotating service account credentials doesn't disturb users, and tracking activity per application becomes much easier. Use a naming pattern like svc-<application>-<role> so ACLs are easy to audit.

#Custom Authorizers

#Implementing a Custom Authorizer

If the built-in policy isn't enough, implement org.apache.kafka.server.authorizer.Authorizer and configure it on the broker:

authorizer.class.name=com.example.ExternalAuthorizer

authorizer.class.name replaces the built-in authorizer. A custom authorizer receives an authorize() call for every request and can decide based on any logic — including calling an external service.

#External Integration and RBAC

A custom authorizer enables integration with centralized policies: authorization databases, RBAC services, or enterprise IAM systems. Principals are mapped to roles, and roles determine access rights. This avoids scattering ACLs across many clusters and unifies policy in one place.

#Closing

In this episode 17 you've understood the concepts of principals, resource types, and ACL operations, managed ACLs with kafka-acls.sh, applied producer and consumer ACL patterns, the least privilege principle, and custom authorizers with RBAC integration.

The key takeaways:

• Without ACLs, all access is denied: default deny.
• Consumers need Read ACLs on both the topic and the group.
• --resource-pattern-type PREFIXED gives a safe wildcard scope.
• Apply least privilege with separate service accounts per application.
• Admin and ClusterAction are tightly restricted to dedicated principals.
• Custom authorizers unify policy with external systems.

In the next episode 18 we'll discuss encryption and TLS/SSL — creating CAs and certificates, keystores and truststores, configuring TLS on brokers and clients, encrypting data in transit, and managing the certificate lifecycle.