Learn Apache Kafka - Cloud-Native Kafka (AWS MSK, Confluent Cloud)

#Introduction

Operating Kafka yourself means managing brokers, upgrades, monitoring, and recovery — a real burden. Managed Kafka moves that burden to the cloud provider: you focus on applications, not infrastructure. Episode 30 compares the main services and shows how to use them.

You'll learn AWS MSK along with serverless and MSK Connect, Confluent Cloud with its tiers, Azure Event Hubs and Aiven as alternatives, and cost, vendor lock-in, multi-cloud, and hybrid strategy considerations.

#Managed Kafka Services

#Service Comparison

Four main services you should know:

• AWS MSK (Managed Streaming for Apache Kafka): native Kafka on AWS, fully compatible with the Kafka API, integrated with IAM and VPC.
• Confluent Cloud: Kafka with the full Confluent ecosystem (Schema Registry, ksqlDB, Connect) as SaaS.
• Azure Event Hubs for Kafka: an endpoint compatible with the Kafka protocol on top of the Event Hubs service.
• Aiven for Apache Kafka: a managed multi-cloud open-source service focused on portability.

#Choosing a Service

Consider: how deeply it integrates with the cloud you use (MSK for AWS), the Confluent ecosystem needs (Schema Registry, ksqlDB, Connect as a service), and multi-cloud needs (Aiven). Protocol compatibility means ordinary Kafka clients work with all these services.

#AWS MSK

#Creating a Cluster

MSK clusters are created via the CLI or console:

aws kafka create-cluster-v2 \
  --cluster-name orders-kafka \
  --provisioned '{"brokerNodeGroupInfo":{"instanceType":"kafka.m7g.large","clientSubnets":["subnet-a"],"clientAuthentication":{"sasl":{"iam":{"enabled":true}}}},"numberOfBrokerNodes":3,"kafkaVersion":"3.7"}'

aws kafka create-cluster-v2 creates a provisioned cluster with three brokers. numberOfBrokerNodes: 3 and kafkaVersion: 3.7 determine the scale; clientAuthentication.sasl.iam enables IAM authentication (without managing separate users).

#Serverless MSK

For fluctuating workloads, MSK Serverless adjusts capacity automatically — you don't set the number of brokers. Suitable for applications with unpredictable traffic or early stages that don't yet know their capacity. Trade-offs: maximum capacity per cluster is limited and per-use cost is higher at large volumes.

#MSK Connect and IAM Authentication

• MSK Connect: a managed service for running Kafka Connect connectors (episode 12) without managing workers.
• IAM authentication: IAM principals replace SASL users; IAM policies control access per action and resource — a clean approach for full AWS environments.

#VPC and CloudWatch

MSK clusters live inside a VPC. Clients outside the VPC need VPC peering, PrivateLink, or a load balancer. Monitoring via CloudWatch exposes key metrics (under-replicated partitions, offline partitions, throughput) without a separate Prometheus setup.

#Confluent Cloud

#Cluster Types and ksqlDB Cloud

Confluent Cloud offers tiers:

• Basic: for development and small loads.
• Standard: general production with security and monitoring features.
• Dedicated: isolated tenancy, guaranteed capacity, for large loads and strict compliance.

ksqlDB Cloud (episode 14) runs as a service, as do the managed Schema Registry and marketplace connectors — you use the Confluent ecosystem without operating servers.

Setting up Confluent Cloud is done via the CLI:

confluent login
confluent environment create dev --resource-group rg-dev
confluent kafka cluster create orders-kafka --cloud aws --region ap-southeast-1 \
  --type basic --environment dev
confluent api-key create --resource orders-kafka

confluent kafka cluster create --type basic creates a Basic tier cluster in the chosen region, and confluent api-key create generates client credentials. The entire cluster lifecycle — including destroy — is managed through the same CLI.

#Key Features

• Stream Lineage: visualization of data origins between topics and pipelines.
• Connectors marketplace: hundreds of ready-to-use connectors.
• Managed Schema Registry: schema governance without operations.
• Cost optimization: monitor credits/units used and right-size clusters.

#Cloud Considerations

#Vendor Lock-in

Using MSK ties you to AWS (IAM, VPC, CloudWatch), while Confluent Cloud is cloud-agnostic but tied to Confluent services. Reduce risk by: using the standard Kafka API (portable), storing topic definitions and schemas as code (episode 31), and designing an abstraction layer in applications.

#Cost Analysis and Multi-Cloud

Calculate the total cost: broker instances, storage, egress network, and additional components (Connect, Schema Registry). Compare with self-managed costs (node rental, engineering hours, outage risk). A hybrid strategy allows on-premise Kafka for sensitive data plus cloud for burst capacity; multi-cloud uses different services per region for redundancy — coordinate with cross-cluster replication (episode 25).

#Closing

In this episode 30 you've understood managed Kafka services: AWS MSK with serverless and MSK Connect, Confluent Cloud with its tiers and ecosystem, Azure Event Hubs and Aiven as alternatives, and cost, vendor lock-in, and multi-cloud strategy considerations.

The key takeaways:

• MSK provides native Kafka integrated with the AWS ecosystem.
• MSK Serverless adjusts capacity for fluctuating traffic.
• IAM authentication replaces SASL users on AWS.
• Confluent Cloud offers managed Schema Registry, ksqlDB, and Connect.
• Event Hubs and Aiven provide Kafka protocol across clouds.
• Calculate total cost of ownership and reduce lock-in with Kafka standards.

In the next episode 31 we'll discuss CI/CD and infrastructure as code — Terraform for Kafka, topic and ACL configuration as code, CI/CD pipelines for testing and canary deployments, and GitOps with ArgoCD and Flux.