Learn Apache Spark - Spark Streaming & Structured Streaming

#Introduction

So far every episode has focused on batch processing — static data that's read, processed, then written. Episode 10 opens up a new dimension: streaming, processing data that keeps flowing in real time. From user logs and IoT telemetry to transaction events, streaming is the backbone of many modern systems.

Spark handles streaming with two approaches: the older DStream and the modern Structured Streaming. They differ fundamentally, and understanding the difference keeps you from writing pipelines with an API that's no longer recommended.

In this episode we'll build a streaming pipeline from scratch, understand how to control when data is processed and how long to wait for incomplete data, and direct results to various sinks including Kafka.

#DStream vs Structured Streaming

#DStream: The Classic Approach

DStream (Discretized Stream) is Spark's earliest streaming abstraction, built on top of RDDs. Data is divided into micro-batches as RDDs that execute separately. Its API is based on RDD transformations and is accessed through streamingContext.

DStream still works, but it's now in maintenance mode. There's no Catalyst optimization, DataFrame integration is limited, and there's no native event-time support. For new projects, the community recommends Structured Streaming.

#Structured Streaming: Modern and Declarative

Structured Streaming treats streaming data as an unbounded table. Each new batch of data is a new row appended to that table. You write the same query as batch processing, and Spark repeats it incrementally.

DStream            → a stream of RDDs, RDD API
Structured Streaming → an unbounded table, DataFrame/SQL API

Its main advantages: one codebase for batch and streaming, Catalyst optimization, event-time windowing, and exactly-once semantics with supporting sinks. This is why Structured Streaming has become the primary choice for new projects.

#Building a Streaming Pipeline

#Initializing the Session and ReadStream

A streaming pipeline starts with an ordinary SparkSession, then reads from the source via readStream:

Reading a stream from Kafka

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
 
spark = SparkSession.builder \
    .appName("streaming-kafka") \
    .config("spark.sql.shuffle.partitions", "8") \
    .getOrCreate()
 
kafka = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "klik-pengguna") \
    .load()

spark.readStream.format("kafka") reads Kafka records. The raw data is in the value column as binary, so it needs to be decoded into a string first.

#Transformations and WriteStream

Decode the data, then run a window aggregation:

Window aggregation and writeStream

from pyspark.sql import functions as F
 
klik = kafka.select(
    F.from_json(F.col("value").cast("string"),
                F.schema_of_json('{"user_id":"1","waktu":"2026-08-10T10:00:00Z","halaman":"/beranda"}'))
    .alias("data")
).select("data.*")
 
per_menit = klik \
    .groupBy(F.window("waktu", "1 minute"), "halaman") \
    .count()
 
query = per_menit.writeStream \
    .outputMode("update") \
    .format("console") \
    .start()

per_menit.writeStream starts a streaming query that runs continuously. outputMode("update") only emits the rows that changed. The query runs in the background; you can call query.awaitTermination() to block until it finishes.

#Trigger Modes, Watermarking, and Output Modes

#Trigger Modes

Triggers determine when a micro-batch is run:

• Default: as fast as possible, each batch finishes and the next one starts immediately.
• Processing time: a fixed interval, e.g. 5 seconds.
• Once: only one batch, then it stops — useful for one-shot jobs.
• Continuous: low latency with certain sinks, experimental.

Processing time trigger of 5 seconds

per_menit.writeStream \
    .trigger(processingTime="5 seconds") \
    .format("console") \
    .start()

#Watermarking

A watermark is the threshold for late data. Because data can arrive late (for example telemetry delayed on the network), the watermark tells Spark how long to wait before considering an event's window complete:

10 minute watermark

per_menit = klik \
    .withWatermark("waktu", "10 minutes") \
    .groupBy(F.window("waktu", "1 minute"), "halaman") \
    .count()

withWatermark("waktu", "10 minutes") makes the aggregation only account for events up to 10 minutes after the window ends. Data arriving past the watermark is dropped — so the stored state doesn't grow without bound.

#Output Modes

Three output modes you need to understand:

• append: only new rows are added — suited to windowing with a watermark.
• update: changed rows are emitted again — suited to aggregations.
• complete: the entire result is rewritten each batch — suited to aggregations without a watermark.

append   → final results (with watermark), file sink
update   → changed results, console sink
complete → the entire aggregation result, small state size

#Sinks to Kafka, Files, Console, and Storage

#Console Sink for Debugging

format("console") is the fastest way to view streaming results in the terminal. Suited for development, not production.

#File Sink for Data Lakes

Writing to Parquet uses the file sink with append mode and a path:

Sink to Parquet

per_menit.writeStream \
    .outputMode("append") \
    .format("parquet") \
    .option("path", "data/hasil_streaming") \
    .option("checkpointLocation", "data/checkpoint") \
    .start()

The checkpointLocation parameter is required — this is where Spark stores processing position metadata so it can resume (recover) after a failure. Checkpoint details are covered in depth in episode 16.

#Kafka Sink

To forward results to other systems, write back to Kafka:

Sink back to Kafka

hasil.writeStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("topic", "hasil-analitik") \
    .option("checkpointLocation", "data/checkpoint-kafka") \
    .start()

The value column must be binary, so don't forget to convert the result to a string and encode it before writing.

#Conclusion

Episode 10 opens the door to real-time processing: you understand why Structured Streaming replaced DStream, build pipelines with readStream and writeStream, control execution with trigger modes, limit late data with watermarking, and direct results to console, file, or Kafka.

Key takeaways:

• Structured Streaming treats a stream as an unbounded table.
• readStream reads sources; writeStream writes results and runs the query.
• Watermarks prevent state from growing with late data.
• Choose the output mode to match the aggregation type: append, update, or complete.
• checkpointLocation is required for recovery and fault tolerance.

In the next episode, episode 11, we'll discuss machine learning with MLlib — the pipeline API with transformers and estimators, feature engineering, model training and tuning, and use cases for classification, regression, clustering, and recommendation.