Learn Apache Kafka - CI/CD & Infrastructure as Code

#Introduction

Kafka clusters configured through manually run CLI commands are a source of inconsistency: no one knows exactly what the production configuration is, and changes are hard to audit. Infrastructure as Code (IaC) moves all definitions — clusters, topics, ACLs, schemas, connectors — into a version-controlled repository.

Episode 31 covers Terraform for Kafka infrastructure, topic and ACL configuration as code, CI/CD pipelines for testing and staged deployment, and GitOps with ArgoCD and Flux for automatic synchronization.

#Infrastructure as Code

#Terraform for Kafka

Terraform can manage cloud Kafka clusters (MSK, Confluent) as well as Kafka resources themselves (topics, ACLs) via providers:

resource "aws_msk_cluster" "orders" {
  cluster_name  = "orders-kafka"
  kafka_version = "3.7.0"
  number_of_broker_nodes = 3
  broker_node_group_info {
    instance_type   = "kafka.m7g.large"
    client_subnets  = [aws_subnet.a.id, aws_subnet.b.id, aws_subnet.c.id]
  }
}

aws_msk_cluster declares the cluster along with its version and instances. Because the definition is in code, every cluster change goes through a pull request and can be reviewed before being applied.

#Kubernetes Manifests and Ansible

• Strimzi (episode 28): clusters, users, and topics are declared as CRD YAML in the repository — a perfect combination with GitOps.
• Ansible: for provisioning self-managed Kafka nodes — repeatable install, server.properties configuration, and service management.
• CloudFormation/ARM: AWS and Azure templates for those already invested in those ecosystems.

#Configuration as Code

#Topic and ACL Definitions

Define topics and ACLs in files that can be reviewed and tested:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
spec:
  partitions: 6
  replicas: 3
  config:
    cleanup.policy: delete
    retention.ms: 604800000

partitions: 6 and config.retention.ms document intent clearly. Version control provides history: when a topic was changed, by whom, and for what reason.

#Schema and Connector Config

• Schemas: store Avro/Protobuf schemas (episode 7) in the repository and register them via the pipeline — schema changes go through review and tested compatibility.
• Connectors: Kafka Connect connector configuration (episode 12) is stored as JSON/YAML and applied via the REST API in the pipeline.

#Version Control Strategy

One repository per service or a monorepo with directories per environment (dev, staging, prod) are common choices. The key point: changes go through pull requests, reviews, and automatic application — so drift between environments can be prevented.

#CI/CD Pipelines

#Automated Testing

The CI pipeline runs tests before changes are applied:

• Unit and integration tests for client applications (episode 32).
• Schema validation: test the compatibility of new schemas against old ones (backward/full).
• Contract testing: make sure producers and consumers understand the same format.
• Performance testing: benchmark the impact of configuration changes before production.

#Canary and Blue-Green

Kafka application deployments can use common deployment patterns:

• Canary: introduce the new version to a fraction of traffic; monitor metrics and lag; if healthy, expand.
• Blue-green: two environments; move all clients from the old version to the new after validation; quick rollback by switching back.

Both are very useful when changing consumer logic or schemas — where errors are better detected on a fraction of traffic than on all of it.

#GitOps Workflows

#ArgoCD and Flux

GitOps makes the git repository the single source of truth. ArgoCD or Flux watch the repository and synchronize state to the cluster automatically:

argocd app sync kafka-stack

argocd app sync kafka-stack pulls the latest definitions from git and applies them. All changes must go through commits; the operator rejects manual changes that deviate from git.

#Declarative Config, Drift, and Rollback

• Declarative configurations: the desired state is fully stated; the operator realizes it.
• Drift detection: ArgoCD flags differences between git state and actual state — triggering investigation or auto-sync.
• Rollback: because every state is in git, rollback is just a revert commit — the operator restores everything to the previous version.

#Closing

In this episode 31 you've understood infrastructure as code with Terraform, topic, ACL, schema, and connector definitions as code, CI/CD pipelines with testing and canary deployments, and GitOps with ArgoCD and Flux.

The key takeaways:

• Cluster, topic, and ACL definitions should be version controlled.
• Terraform and Strimzi CRDs are the main ways to declare infrastructure.
• Test schemas and contracts in the pipeline before deployment.
• Canary and blue-green reduce the risk of client changes.
• GitOps makes git the single source of truth.
• Drift detection and automatic rollback keep production consistent.

In the next episode 32 we'll discuss testing strategies — unit tests for producers, consumers, and Streams topologies, integration tests with Testcontainers and embedded Kafka, performance tests with kafka-producer-perf-test, and chaos engineering for testing failures.