Learn Apache Kafka - Docker & Container Best Practices

#Introduction

Docker is the fastest way to run Kafka in development and often in production too. The official image already packages Kafka, KRaft, and all the CLI tools — you just configure things through environment variables. But containers have traps: JVMs unaware of memory limits, misdirected listeners, and data lost because volumes aren't mounted.

Episode 29 covers the official and Confluent images, custom Dockerfiles for client applications, Docker Compose for a complete stack — multi-broker, Schema Registry, ksqlDB — and volume, JVM, health check, and logging considerations inside containers.

#Kafka Docker Images

#Official and Confluent Images

Two main image sources:

• apache/kafka: the official Apache Kafka image, supporting KRaft and environment variables.
• confluentinc/cp-kafka: Confluent images with ecosystem utilities (Schema Registry, ksqlDB) and complete env configuration.

For development, apache/kafka is enough; for a Confluent stack (Schema Registry, Connect, ksqlDB), the Confluent images integrate more easily.

#Environment Variable Configuration

Kafka images accept broker configuration via environment variables with the KAFKA_ pattern:

KAFKA_BROKER_ID: "1"
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_NODE_ID: "1"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"

KAFKA_PROCESS_ROLES: broker,controller runs the combined roles (KRaft), and KAFKA_ADVERTISED_LISTENERS sets the announced address — the part most often gotten wrong in containers (episode 15).

#Custom Dockerfiles

#Best Practices for Client Applications

Client application images should be lightweight and declarative. Main principles: applications and Kafka are separate, images use pinned base versions, and configuration is injected at runtime:

FROM eclipse-temurin:17-jre-alpine
COPY target/order-producer.jar /app/app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app/app.jar"]

eclipse-temurin:17-jre-alpine is a lightweight JRE base. The same principles apply to all languages: separate build artifacts, minimal base images, secrets not baked into the image.

#Docker Compose Setup

#Complete Multi-Broker Stack

Compose combines the whole stack in one file. An example single broker with Schema Registry and ksqlDB:

services:
  broker:
    image: confluentinc/cp-kafka:7.6.0
    ports: ["9092:9092"]
    environment:
      KAFKA_BROKER_ID: "1"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
  schema-registry:
    image: confluentinc/cp-schema-registry:7.6.0
    ports: ["8081:8081"]
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: broker:29092

PLAINTEXT://broker:29092 is used between services, PLAINTEXT_HOST://localhost:9092 for clients from the host — the separation covered in episode 15. Port 8081 exposes the Schema Registry.

#Networking Between Services

Use the default Compose network: services call each other by service name (broker, schema-registry). For production, define explicit networks and restrict connections; for development, ports exposes the HOST listener to your machine.

#Container Considerations

#Volumes and JVM Settings

Broker data must survive container restarts:

volumes:
  - kafka-data:/var/lib/kafka/data
deploy:
  resources:
    limits:
      memory: 2g
      cpus: "2"

volumes: kafka-data:/var/lib/kafka/data stores logs in a named volume. Without a volume, all data is lost when the container is removed — a disaster for production. Also point the log location to the mounted directory (log.dirs).

#Memory Limits and Heap

The classic problem: the JVM reads host RAM and sets the heap too large, or doesn't recognize the container memory limit. The solution is a container-aware JVM:

java -Xms512m -Xmx1g -jar kafka-server-start.jar config/server.properties

-Xmx1g explicitly locks the heap. Make sure limits.memory leaves room above the heap for off-heap, page cache, and overhead. Don't let the JVM think the full host RAM is available.

#Health Checks

Containers need health checks so orchestrators know when they're ready:

healthcheck:
  test: ["CMD-SHELL", "kafka-broker-api-versions.sh --bootstrap-server localhost:9092"]
  interval: 10s
  timeout: 5s
  retries: 3

kafka-broker-api-versions.sh --bootstrap-server localhost:9092 verifies the broker actually responds, not just that the process is alive. This gives a reliable signal for orchestration readiness.

#Logging Drivers

Containers should write logs to stdout/stderr so logging drivers (json-file, journald, or sidecars) capture them. Don't write logs to files inside the container without collection. Limit the log driver size so a log explosion doesn't fill the host disk.

#Closing

In this episode 29 you've understood containerizing Kafka: official and Confluent images, environment variable configuration, custom Dockerfiles for applications, Docker Compose for a complete stack, and volume, JVM, memory, health check, and logging considerations.

The key takeaways:

• Use official images and configure via environment variables.
• Advertised listeners must differ for containers and the host.
• Mount a volume for broker data; without it, data is lost on restart.
• Set -Xmx explicitly; the JVM isn't automatically aware of container limits.
• Health check with the Kafka CLI, not just a process check.
• Direct logs to stdout/stderr and limit their size.

In the next episode 30 we'll discuss cloud-native Kafka — AWS MSK, Confluent Cloud, and Azure Event Hubs. You'll learn creating an MSK cluster, IAM authentication, MSK Connect, Confluent Cloud tiers, and cost and vendor lock-in considerations.