Learn Apache Kafka - Kubernetes Deployment with Strimzi

#Introduction

Running Kafka on Kubernetes manually — StatefulSets, services, TLS secrets, and rolling updates — is error-prone work. Strimzi provides a Kubernetes Operator that automates all of it: you describe the cluster with a Custom Resource, and the operator realizes it along with its maintenance operations.

Episode 28 covers the operator and CRD pattern, the main Strimzi resources (Kafka, KafkaConnect, KafkaUser), how StatefulSets and storage work, automatic TLS and ACL management, and production considerations like anti-affinity, storage classes, and multi-zone.

#The Strimzi Kafka Operator

#Kubernetes Operator Pattern

An operator is an application that handles the lifecycle of custom resources: it watches declarations (CRDs), realizes them into K8s resources, and continuously reconciles actual state with desired state. Strimzi contains several operators: the Cluster Operator (brokers, Connect, MirrorMaker), the Topic Operator (topics), and the User Operator (users and ACLs).

#Strimzi CRDs

The main resources you write:

• Kafka: defines the cluster — brokers, controllers, storage, and listeners.
• KafkaConnect: a Kafka Connect worker cluster.
• KafkaTopic: declarative topic definitions.
• KafkaUser: users and their ACLs.
• KafkaBridge: a REST bridge for non-Kafka clients.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    version: 3.7.1
    storage:
      type: persistent-claim
      size: 100Gi
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true

spec.kafka.replicas: 3 declares three brokers, and storage: persistent-claim requests a PersistentVolume per pod. The operator creates the StatefulSet, services, and secrets according to this declaration.

#Deploying on K8s

#StatefulSets for Brokers

Kafka brokers run as a StatefulSet because every pod needs a stable identity (pod name = broker identity) and separate storage. Strimzi creates one pod per broker with its own volume. Rolling updates are done one by one by the operator — brokers never go down at the same time.

#PersistentVolume Management

Storage is declared per cluster:

storage:
  type: persistent-claim
  size: 100Gi
  class: gp3
  deleteClaim: false

class: gp3 selects a storage class (for example EBS gp3 on AWS); deleteClaim: false keeps volumes around when a cluster is deleted — preventing accidental data loss. Choose a storage class with enough IOPS for Kafka writes.

#Services and Configuration

Strimzi creates services for broker access: one for internal pod-to-pod communication, and one per listener for clients. Broker configuration (for example log segments, retention) is set in spec.kafka.config. TLS credentials and certificates are stored as Secrets automatically mounted into broker pods.

#Strimzi Features

#Rolling Updates and TLS Automation

The operator handles rolling updates when the broker version or configuration changes, including partition maintenance during the process. The most valuable part: automatic TLS management — Strimzi creates the CA, issues and rotates certificates for brokers and clients, so you don't manage certificates manually like in episode 18.

#Users and ACLs via CRDs

Access control is declared, not done with the CLI:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: app-producer
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: orders
          patternType: literal
        operation: Write

authentication.type: tls gives the user a certificate for mutual TLS, and authorization.acls replaces kafka-acls.sh — the operator applies the ACLs to the cluster automatically.

#Monitoring and Cruise Control

Strimzi exposes metrics for Prometheus (episode 22) and can integrate Cruise Control (episode 20) for automatic rebalancing — both are simply enabled via CRD declarations.

#Production Considerations

#Pod Anti-Affinity

One broker per physical node so a node failure doesn't take down several brokers at once:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            strimzi.io/name: my-cluster-kafka

podAntiAffinity prevents two broker pods from being scheduled on the same node. Combine with topology.kubernetes.io/zone to spread brokers across AZs (rack awareness, episode 24).

#Storage Class, Network Policy, and Multi-Zone

• Storage class: choose a volume provider offering consistent IOPS (gp3, io2) for production.
• Network policies: restrict network access between namespaces; only specific application pods may reach broker listeners.
• Multi-zone: use topology spread constraints and broker.rack based on zone labels so replicas spread across AZs.
• Disaster recovery: think about MirrorMaker and volume recovery; back up volumes periodically for extra protection.

#Closing

In this episode 28 you've understood deploying Kafka on Kubernetes with Strimzi: the operator and CRD pattern, StatefulSets and storage, automatic TLS and user management, and production considerations like anti-affinity, storage classes, and multi-zone.

The key takeaways:

• Strimzi automates Kafka's lifecycle through CRDs and operators.
• Brokers run as StatefulSets with a PersistentVolume per pod.
• Strimzi manages CAs, certificates, users, and ACLs automatically.
• deleteClaim: false prevents data loss when a cluster is deleted.
• Anti-affinity spreads brokers across different nodes and zones.
• Monitoring and Cruise Control are enabled via CRD declarations.

In the next episode 29 we'll discuss Docker and container best practices — official and Confluent images, Docker Compose for multi-broker setups with Schema Registry and ksqlDB, and JVM, memory, health check, and volume considerations in containers.