Learn Apache Spark - Real-world Use Cases & Patterns

#Introduction

After 19 episodes of theory and practice, episode 20 brings it all together: how Spark is used to solve real business problems. Three major use cases will be broken down — ETL, real-time analytics, and recommendation systems — along with the architectural patterns that keep pipelines reliable.

This is the episode that connects technical skill with business value. Correct code matters, but what matters more is code that solves problems — and the problems you choose well will determine the quality of the entire data platform.

This episode covers four topics: ETL pipelines, real-time analytics, recommendation systems, design patterns for reliability, and end-to-end batch-plus-streaming architectures.

#Use Case: ETL Pipelines

#Daily ETL to the Data Lake

The most common example: pull data from a transactional database every night, clean it, transform it, and write it to the data lake for analytics:

ETL from database to Parquet

from pyspark.sql import functions as F
 
orders = spark.read \
    .format("jdbc") \
    .option("url", os.environ["DB_URL"]) \
    .option("dbtable", "orders") \
    .option("partitionColumn", "id") \
    .option("lowerBound", 0) \
    .option("upperBound", 1000000) \
    .option("numPartitions", 8) \
    .load()
 
bersih = orders.filter(F.col("amount").isNotNull()) \
    .withColumn("order_date", F.to_date("created_at")) \
    .repartition(4, "order_date")
 
bersih.write \
    .mode("overwrite") \
    .partitionBy("order_date") \
    .parquet("s3a://datalake/orders")

The pattern above uses partitioning on the read side (partitionColumn) and the write side (partitionBy) — both keep the database query from being overloaded and keep the data lake files organized.

#Idempotency in ETL

For batch ETL, make sure every run produces the same result without depending on previous runs:

• Write to a temporary location, then rename to the final location atomically.
• Use mode("overwrite") for partitions that are fully recomputed.
• Consider Delta or Iceberg for truly safe transactions.

#Use Case: Real-Time Analytics

#Product Metrics Dashboard

Real-time analytics answers questions like "how many clicks in the last 5 minutes" or "conversion per channel right now." The architecture: events enter Kafka, Spark aggregates with windows, and results go to a fast sink:

Real-time per-minute aggregation

from pyspark.sql import functions as F
 
events = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "clicks") \
    .load() \
    .select(F.from_json(F.col("value").cast("string"),
            F.schema_of_json('{"halaman":"/","waktu":"2026-08-10T10:00:00Z"}')).alias("d")) \
    .select("d.*")
 
metrik = events \
    .withWatermark("waktu", "2 minutes") \
    .groupBy(F.window("waktu", "1 minute"), "halaman") \
    .count()
 
metrik.writeStream \
    .outputMode("update") \
    .format("console") \
    .start() \
    .awaitTermination()

This pipeline can be directed to a high-speed sink like Cassandra, Redis, or another Kafka topic that feeds a dashboard. Total latency from event to dashboard is usually within seconds.

#Latency Considerations

Choose triggers and sinks to match your needs: processingTime("5 seconds") is enough for a dashboard; for stricter near-real-time needs, consider an engine like Flink. Spark wins this case when the team needs one engine for both batch and stream at the same time.

#Use Case: Recommendation Systems

#Collaborative Recommendation

Recommendation systems use user-product interaction data to predict preferences. With ALS from episode 11, the complete pipeline:

Complete recommendation pipeline

from pyspark.ml.recommendation import ALS
from pyspark.ml.evaluation import RegressionEvaluator
 
(train, test) = interaksi.randomSplit([0.8, 0.2], seed=42)
 
als = ALS(userCol="user_id", itemCol="produk_id", ratingCol="rating",
          coldStartStrategy="drop")
model = als.fit(train)
 
prediksi = model.transform(test)
evaluator = RegressionEvaluator(
    metricName="rmse", labelCol="rating", predictionCol="prediction")
print("RMSE:", evaluator.evaluate(prediksi))
 
rekomendasi = model.recommendForAllUsers(10)

recommendForAllUsers(10) produces 10 recommendations per user. The results can be written to a cache to serve a recommendation API, or recomputed every night with a batch ETL.

#Additional Production Features

Production recommendations usually combine several signals: popularity, categories, and toxicity filters. Spark gives you the flexibility to combine ALS scores with other business logic in a single pipeline before writing the final result.

#Design Patterns for Pipeline Reliability

#Proven Patterns

Several design patterns keep pipelines reliable:

• Checkpoint + transactional sink: the combination for streaming that doesn't lose data.
• Data quality gate: validation before and after transformations; stop on anomalies.
• Idempotent writes: overwrite or merge, not blind appends.
• Retry with backoff and a dead-letter queue: failed jobs are isolated, not blocked forever.
• Schema registry: schema changes are detected before breaking downstream.

ingest → initial validation → transform → final validation → publish → alert on failure

#Monitoring Business Metrics and Data Quality

Reliability is incomplete without observability:

• Business metrics: number of processed rows, key aggregate values, data freshness.
• Data quality: null ratios, value distributions, duplicates, and consistent schemas.
• Every pipeline writes these metrics to an observation table, and alerts fire when numbers deviate from baseline.

#End-to-End Architecture for Batch + Streaming

#One Platform, Two Speeds

An architecture that combines batch and streaming with a single engine:

Kafka/DB → ingestion → Spark (stream) → lakehouse (Delta/Iceberg)
                              ↓
               Spark (batch) → warehouse → BI & ML

This pattern is often called a lightweight lambda architecture: the real-time flow provides fresh data, batch maintains accuracy and completeness, and the lakehouse is the meeting point of both. Spark becomes the single engine for both.

#Design Considerations

• Medallion architecture: bronze (raw), silver (clean), gold (aggregated) — separating the data layers that can be accessed.
• One schema, one platform: avoid many copies of data with different schemas.
• Cost control: batch for large, non-urgent data; streaming for what needs speed.

#Conclusion

Episode 20 shows Spark in real action: ETL connects databases to data lakes, real-time analytics serves fresh metrics, recommendation systems use MLlib end to end, design patterns maintain reliability, and batch-plus-streaming architecture uses Spark as a single platform.

Key takeaways:

• Good ETL is idempotent and uses two-way partitioning.
• Real-time analytics flows from Kafka through windows to a fast sink.
• ALS completes the recommendation pipeline with RMSE evaluation and persistence.
• Quality gates and data quality monitoring keep data trustworthy.
• A lakehouse lets batch and streaming share one storage layer.

In the next episode, episode 21, we'll discuss the ecosystem and resources — tooling like Delta Lake, Iceberg, Hudi, and the Spark SQL Gateway, managed Spark services like Databricks, AWS EMR, and GCP Dataproc, libraries and extensions like GraphX and the Pandas API on Spark, plus learning resources from the community and official documentation.