Learn Apache Kafka - Producers: Writing Data to Kafka

#Introduction

The producer is Kafka's write side — the application that turns business facts into events sent to topics. Although it looks simple, the producer holds many important decisions: how long to wait for broker confirmation (acks), how many retries, how big the batches should be, and whether data can be delivered as duplicates.

Episode 5 takes you from zero to proficiency in writing producers: understanding the ProducerRecord structure, the producer API in four popular languages, the fire-and-forget, synchronous and asynchronous delivery patterns with callbacks, and tuning the parameters that determine throughput and durability.

By the end of this episode, you'll understand the idempotent producer — the key toward exactly-once semantics that we'll dig into in episodes 8 and 9.

#Producer Basics

#Basic Configuration

Every producer needs at minimum bootstrap.servers=localhost:9092 — the list of initial brokers for finding the cluster. Other important parameters include key.serializer and value.serializer, which turn objects into bytes:

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
acks=all

#ProducerRecord

The unit of delivery is the ProducerRecord, which contains a topic, an optional partition, a key, a value, a timestamp, and headers. If the partition isn't provided, the partitioner determines the destination based on the key; if the key is also empty, records are spread round-robin or sticky.

#Synchronous vs Asynchronous

Sending a record to Kafka is asynchronous internally: the producer places records into a buffer, then sends batches to the broker in the background. send() returns a future that can be blocked on (get()) for synchronous behavior, or given a callback to receive the result without blocking.

#Producer APIs in Different Languages

#Java Producer API

Java is Kafka's native language. A minimal example:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
 
try (Producer<String, String> producer = new KafkaProducer<>(props)) {
    producer.send(new ProducerRecord<>("orders", "order-001", "created"));
}

The try-with-resources pattern ensures the producer is closed, flushing any remaining buffer before exiting.

#Python, Go, and Node.js

The main ecosystems outside Java:

• Python: confluent-kafka (librdkafka bindings) or kafka-python.
• Go: confluent-kafka-go or Shopify's sarama.
• Node.js: kafkajs, a popular library with a modern promise-based API.

confluent-kafka Python producer

from confluent_kafka import Producer
 
p = Producer({"bootstrap.servers": "localhost:9092"})
p.produce("orders", key="order-001", value="created")
p.flush()

All libraries follow the same model: configure, send, and flush/poll to make sure batches are delivered.

#Message Sending Patterns

#Fire-and-Forget

The simplest pattern: call send() without caring about the result. It's suitable for data that can be lost if the broker has problems, but it's usually unwise because errors like a missing topic are silently missed.

#Synchronous Send

Blocks until the broker responds. The safest for correctness, but it sacrifices throughput because every send waits for a round-trip:

producer.send(record).get();

#Asynchronous Send with Callback

The best pattern for both throughput and error awareness: send() doesn't block, and the callback runs when the result arrives:

producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        System.err.println("Failed to send: " + exception.getMessage());
    } else {
        System.out.println("OK to partition " + metadata.partition());
    }
});

#Batching Messages

Instead of sending one by one, the producer collects records into batches. Two parameters govern batching: batch.size (maximum bytes per batch, default 16KB) and linger.ms (how long to wait before sending, default 0). Raising linger.ms=20 often dramatically increases throughput at the cost of a small latency increase.

#Producer Configuration Tuning

#Acks and Retries

acks determines the level of confirmation:

• acks=0: no confirmation, fastest, prone to loss.
• acks=1: leader writes to its local log, without waiting for followers.
• acks=all (or -1): all ISRs confirm, highest durability.

retries controls how many times a failed send is retried, and retry.backoff.ms is the pause between attempts. With acks=all and idempotence, retries are safe from duplicates.

#In-Flight Requests and Compression

max.in.flight.requests.per.connection limits how many requests can be unacknowledged. A value above 1 can create reordering if retries happen, unless idempotence is enabled. compression.type (gzip, snappy, lz4, zstd) compresses the payload: zstd offers the best ratio, suitable for large payloads.

batch.size=65536
linger.ms=20
compression.type=zstd
max.in.flight.requests.per.connection=5
buffer.memory=33554432

buffer.memory limits the producer's total memory for records waiting to be sent; if it's full, send() will block until space is available.

#Idempotent Producers

#Exactly-Once Semantics Setup

Without protection, a retry can send a record twice if the first confirmation is lost. The idempotent producer eliminates this duplication by giving each producer a producer ID and a sequence number per partition:

enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5

When enable.idempotence=true, the broker automatically forces acks=all and a large retries. This combination guarantees no duplicates and no reordering per partition — the foundation of exactly-once writes.

#Transactional ID

For cross-partition atomicity and end-to-end stream processing, use the transactional producer with transactional.id. This value must be unique and stable per instance. Full details will be covered in episode 9:

transactional.id=order-svc-1
transaction.timeout.ms=60000

#Closing

In this episode 5 you've written producers in four languages, understood the ProducerRecord structure, the three delivery patterns (fire-and-forget, synchronous, asynchronous with callback), batching techniques, and tuned acks, retries, max.in.flight.requests, compression.type, batch.size, and linger.ms. Finally, you got to know the idempotent producer as the first step toward exactly-once.

The key takeaways:

• A ProducerRecord contains a topic, key, value, timestamp, and headers.
• Asynchronous sends with callbacks are the best pattern for throughput and error awareness.
• acks=all gives the highest durability; acks=0 is the fastest but prone to data loss.
• Batching with linger.ms and batch.size significantly increases throughput.
• compression.type=zstd is suitable for large payloads.
• enable.idempotence=true prevents duplicates and reordering during retries.

In the next episode 6 we reverse direction: consumers — how to read data from Kafka. You'll learn subscription, polling, consumer groups and rebalancing, offset management with manual commit and auto-commit, tuning fetch.min.bytes and max.poll.records, and correct consumer patterns so you don't stall. Get your first consumer code ready!