Learn Apache Kafka - Network Configuration & Multi-Datacenter

#Introduction

The most common mistake in Kafka setups isn't on the application side, it's in the network: clients that can't connect, high inter-broker latency, or communication failing across environments. Most of it comes from a weak understanding of listeners and advertised listeners.

In episode 15 you'll understand how Kafka configures connection endpoints, distinguish the PLAINTEXT, SSL, and SASL protocols, set up separate internal and external listeners, and explore multi-datacenter deployment patterns — active-active and active-passive — along with their latency and recovery considerations.

#The Listener Concept

#Listener and Advertised Listener

Every broker opens one or more listeners: a combination of host, port, and protocol where the broker accepts connections. When a broker responds to a client, it doesn't refer to itself by the address the client used to connect, but by the advertised listener — the address it announces so clients can reach it again.

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka-1.internal:9092

If advertised.listeners is wrong, clients successfully send their first metadata request but fail to reach the broker because the announced address is unreachable — the classic "connection timeout on the client even though the broker is healthy" symptom.

#Listener Protocols

The protocol determines how a connection is secured:

• PLAINTEXT: no encryption or authentication, development only.
• SSL: connection encrypted with TLS (details in episode 18).
• SASL_PLAINTEXT: SASL authentication without encryption.
• SASL_SSL: SASL authentication plus TLS encryption — the standard production combination.

#Inter-Broker Communication

Communication between brokers is governed by inter.broker.listener.name:

listeners=INTERNAL://:9093,EXTERNAL://:9094
advertised.listeners=INTERNAL://10.0.1.5:9093,EXTERNAL://kafka.example.com:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SASL_SSL
inter.broker.listener.name=INTERNAL

inter.broker.listener.name=INTERNAL makes replication between brokers run over the internal network without crossing the internet, while external clients use the EXTERNAL listener secured with SASL_SSL. The listener.security.protocol.map value maps listener names to their actual protocols.

#Multiple Listeners and Networking

#Internal vs External

Separating internal and external listeners is a basic security practice: clients inside the VPC use private addresses, public clients use secured public addresses, and both share the same brokers. This avoids exposing the entire cluster to the internet while keeping replication throughput on the internal network.

#Docker Networking

In Docker, brokers are often unreachable because the advertised listener uses a hostname only known inside the container. Use the Compose service name for fellow containers and a different hostname for host clients:

KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

PLAINTEXT://kafka:29092 is used between containers, while PLAINTEXT_HOST://localhost:9092 is for clients from the host. Full configuration is covered in episode 29.

#Cloud Networking

On AWS, brokers sit in a VPC with security groups that only allow broker ports from specific CIDRs. For clients outside the VPC, put a load balancer or AWS PrivateLink in front of the listener. Always restrict the security group source IPs to internal networks and trusted principals.

#DNS Considerations

Advertised listeners use hostnames, so DNS must be consistent from both the client's and brokers' points of view. An important principle: the name announced by a broker must resolve to the correct address from where the client is. Don't use bare IP addresses if brokers can be moved or scaled, because IPs can change — use stable DNS names and let the resolver point to the right address.

kafka-1.internal.example.com   -> 10.0.1.5
kafka-2.internal.example.com   -> 10.0.1.6
kafka.example.com              -> public load balancer

kafka.example.com points to the load balancer for external clients, while the kafka-N.internal.example.com names are used among brokers and internal clients. Also make sure reverse lookup works if mutual TLS and hostname-based principal mapping are used.

#Multi-Datacenter Patterns

#Cross-Datacenter Replication

Kafka doesn't replicate data across datacenters natively — replication only happens between brokers in a single cluster. For multi-datacenter, you use cross-cluster replication patterns, especially MirrorMaker 2.0, covered in depth in episode 25.

Before choosing a pattern, set the business target: how much data can be lost and how fast recovery needs to be. These targets (RPO and RTO) determine how synchronous replication must be and how automated the failover is.

#Active-Active vs Active-Passive

• Active-active: both datacenters accept writes independently; data is synchronized in both directions. Higher throughput, but prone to conflicts and needs careful offset translation.
• Active-passive: one datacenter accepts writes, the other stands by; replication is one-way. Simpler and safer, with the trade-off that failover must wait for synchronization.

#Latency and Disaster Recovery

Physical distance determines latency: cross-continent replication can add hundreds of milliseconds per round-trip, lowering production throughput and slowing failover. For a strict RTO (Recovery Time Objective), place datacenters close together and design automated failover with routine checkpoints.

#Closing

In this episode 15 you've understood the difference between listeners and advertised listeners, the four connection protocols, separating internal and external listeners, Docker and cloud networking considerations, and the active-active and active-passive multi-datacenter patterns.

The key takeaways:

• The advertised listener is the address announced by the broker; misconfiguration makes clients time out.
• The standard production protocol: SASL_SSL for authentication plus encryption.
• Separate internal listeners for replication and external listeners for clients.
• In Docker, advertised listeners must differ for containers and the host.
• Multi-datacenter uses cross-cluster replication, not within-cluster replication.
• Active-passive is safer; active-active is faster but prone to conflicts.

In the next episode 16 we'll secure access with SASL authentication — PLAIN, SCRAM, GSSAPI, and OAUTHBEARER. You'll learn broker and client configuration, creating SCRAM users with kafka-configs, and OAuth 2.0 integration.