Learn Apache Kafka - Testing Strategies

#Introduction

Code that uses Kafka can feel hard to test: it needs brokers, asynchronous timing, and external dependencies. As a result many teams skip testing and pay for it with production bugs. Yet there are layered strategies that make testing Kafka easy and meaningful.

Episode 32 covers four layers of testing: unit tests for producer, consumer, and Streams topology logic; integration tests with embedded Kafka and Testcontainers; performance tests with the official tools; and chaos engineering for testing behavior during real failures.

#Unit Testing

#Testing Producers and Consumers

Unit tests focus on logic, not on Kafka. For producers, mock the client:

MockProducer<String, String> mock = new MockProducer<>(
    true, new StringSerializer(), new StringSerializer());
 
OrderProducer producer = new OrderProducer(mock);
producer.sendOrder("order-001");
 
List<ProducerRecord<String, String>> history = mock.history();
assertEquals("orders", history.get(0).topic());

MockProducer captures records without needing a broker, so you can quickly verify the topic, key, and value. For consumers, the same pattern uses MockConsumer to simulate polling and offsets.

#Testing Kafka Streams Topologies

Kafka Streams provides the TopologyTestDriver to run topologies deterministically:

TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props);
TestInputTopic<String, String> input =
    driver.createInputTopic("orders", new StringSerializer(), new StringSerializer());
input.pipeInput("order-001", "PAID");
 
TestOutputTopic<String, String> output =
    driver.createOutputTopic("paid-orders", new StringDeserializer(), new StringDeserializer());
assertEquals("PAID", output.readValue());

TopologyTestDriver runs the topology in memory without Kafka — time can be advanced virtually, and output is verified directly. This is the fastest way to test stream processing logic.

#Mocking Kafka

For non-Java applications or general needs, mocking frameworks (Mockito, or libraries like kafka-node-test-utils) replace the Kafka client. The principle is the same: verify your code's behavior without real infrastructure.

#Integration Testing

#Embedded Kafka

To ensure real interaction with the Kafka protocol, use embedded Kafka: a broker running inside the test process. Libraries like kafka-junit or spring-kafka-test start a broker in the background during tests and shut it down afterward. Suitable for fast test suites without Docker.

#Testcontainers

Testcontainers runs Kafka inside Docker during tests:

docker run -p 9092:9092 apache/kafka:3.7.1

Conceptually, Testcontainers does the above programmatically: the container starts, the test application connects to localhost:9092, then the container is cleaned up automatically. Its advantages: the Kafka version matches production exactly, and the whole stack (broker + Schema Registry) can be tested together.

#Test Fixtures and Schema Tests

• Fixtures: provide realistic initial data for each test, from topics and offsets to state stores.
• Schema compatibility tests: register new schemas with the Schema Registry and test that old versions remain compatible (backward/full) before production use.

#Performance Testing

#kafka-producer-perf-test and Consumers

The official benchmarking tools:

bin/kafka-producer-perf-test.sh --topic orders \
  --num-records 1000000 --record-size 1024 \
  --throughput -1 --producer-props bootstrap.servers=localhost:9092

kafka-producer-perf-test.sh sends 1 million 1KB records with no throughput limit, then reports records/sec, MB/sec, and latency percentiles. This is an honest baseline for evaluating episode 21 tuning.

#Benchmarking and Latency Measurement

Good methodology: measure a baseline, change one parameter, measure again. Report percentiles (p50, p99, p999) — averages are misleading for latency. Run against a realistic cluster (not a single broker) and long enough to reach a steady state.

#Chaos Engineering

#Broker Failure Scenarios

Chaos testing examines behavior during real failures. Common scenarios:

• Broker failure: stop one broker; verify leader election, unavailability duration, and recovery.
• Network partition: isolate a broker from the network; observe effects on ISR and writes.
• Slow consumer: make a consumer slow; observe lag and rebalance.
• Rebalance storm: restart many consumers at once; observe whether rebalances repeat (storm) and the latency impact.

#Safe Chaos Practices

Start in staging, not production. Use tools like Chaos Monkey or simple scripts that kill broker pods (episode 28). Make sure monitoring (episode 22) captures every phase, and define success metrics before starting: for example, write unavailability doesn't exceed X seconds and no data is lost.

#Closing

In this episode 32 you've understood layered testing strategies: unit tests with mocking and the TopologyTestDriver, integration tests with embedded Kafka and Testcontainers, performance tests with the official tools, and chaos engineering for real failures.

The key takeaways:

• Unit tests verify logic with mocks, without a broker.
• The TopologyTestDriver tests Streams topologies deterministically.
• Testcontainers tests real interaction with the production Kafka version.
• kafka-producer-perf-test gives throughput and latency baselines.
• Measure percentiles, not averages, for latency.
• Chaos tests verify failover and rebalance in a safe environment.

In the next episode 33 we'll discuss security best practices and compliance — hardening with least privilege and network segmentation, GDPR compliance with retention and audit logs, security monitoring, and secure development with secret management and scanning.