Learn Apache Spark - History, Background & Why Choose Spark

#Introduction

Episode 0 made sure your environment is ready. Now it's time to understand why Spark exists. Episode 1 answers three big questions: where Spark came from, what its advantages are, and why you — as an engineer working with large-scale data — need it.

Many people start using Spark because of a tutorial, but understanding its background is far more valuable. When you know the problem it solves, you'll find it easier to decide when to use Spark, when to choose Flink, and how to position it in your data architecture. Let's start from the beginning of the story.

#The Evolution from MapReduce and the Hadoop Ecosystem

#The Birth of Hadoop MapReduce

Spark's story begins with Hadoop MapReduce, a distributed data processing paradigm popularized by Google and implemented as open source by Apache Hadoop around 2006. The big idea was revolutionary: write map and reduce functions, and the framework splits data across many machines, runs computations in parallel, then combines the results.

But MapReduce had a structural weakness. Each stage writes its intermediate results to disk before the next stage begins. For complex jobs with many stages, this means repeated disk reads and writes — very slow for iterative workloads like machine learning.

#The Birth of Apache Spark at UC Berkeley

In 2009, researchers at the UC Berkeley AMPLab started the project that would become Apache Spark. Its vision: process data in-memory — storing data in RAM between stages instead of writing to disk — so iterative and interactive workloads could run tens to hundreds of times faster.

The project was researched openly, became a top-level Apache project in 2014, and has since become one of the most active big data projects in the world. In 2016, the community introduced Spark SQL, Structured Streaming, and MLlib, unified in a single API — this is what distinguishes Spark from its predecessors.

#The Catalyst: Hadoop Was Too Heavy for Interactivity

Another important piece of context: the Hadoop ecosystem required many separate components. For batch you needed MapReduce, for SQL you needed Hive, for streaming you needed Storm, for machine learning you needed Mahout. Spark offers one engine for everything: batch, SQL, streaming, graph, and machine learning in a single runtime.

Hadoop era:  MapReduce + Hive + Storm + Mahout (many components)
Spark era:   Spark Core + Spark SQL + Structured Streaming + MLlib (one runtime)

#The Advantages of Apache Spark

#In-Memory Computation

The fundamental difference from MapReduce: Spark stores raw data and intermediate results in memory with the lazy evaluation pattern and a DAG scheduler. Results are not written to disk unless requested. For iterative workloads like training machine learning models that read a dataset repeatedly, this produces dramatic speedups.

#Unified Batch and Stream API

Spark uses the micro-batch model for streaming: incoming data is split into small batches and then processed with exactly the same API as batch processing. That means you write one skill set and use it for both daily ETL and real-time pipelines. This contrasts with other frameworks that force you to learn two different paradigms.

#Ease of Use and Programming Languages

Spark provides concise high-level APIs in Scala, Java, Python, and R. For example, reading CSV, filtering, and aggregating takes just a few lines:

Simple ETL with PySpark

from pyspark.sql import SparkSession
 
spark = SparkSession.builder.appName("contoh").getOrCreate()
df = spark.read.csv("data/transaksi.csv", header=True)
ringkasan = df.groupBy("kategori").sum("jumlah")
ringkasan.show()

df.groupBy("kategori").sum("jumlah") aggregates data across the entire cluster in one line — something that takes dozens of lines in MapReduce. All 23 episodes of this series will build the habit of using this high-level API.

One principle to hold from the start: the higher the level of abstraction you use, the more automatic optimization Spark performs. Start with DataFrames and Spark SQL, then drop down to RDDs only when you truly need low-level control — this is the pattern recommended by almost all data engineering practitioners.

#Comparison with MapReduce, Flink, and Beam

#Spark vs Hadoop MapReduce

• Latency: MapReduce writes intermediate results to disk; Spark stores them in memory. For multi-stage jobs, Spark is far faster.
• API: MapReduce is low-level and verbose; Spark has a concise DataFrame API.
• Fault tolerance: both are strong; MapReduce uses recomputation from disk, Spark uses lineage for recomputation.

#Spark vs Apache Flink

Flink excels at true streaming — processing events one at a time (record-at-a-time) with millisecond latency and very mature state management. Spark uses micro-batches that add tens of milliseconds of latency but are simpler. A rule of thumb: if you need very strict exactly-once semantics and the lowest latency for pure streaming, Flink is a strong candidate; for mixed batch and stream workloads, Spark is more practical.

#Spark vs Apache Beam

Beam is an abstraction (SDK + runner), not an engine. You write a pipeline once and run it on various runners: Dataflow, Flink, or Spark. Spark, by contrast, is a complete engine. Beam is useful when an organization wants cross-engine portability; Spark offers feature depth within its own engine.

#Real-World Use Cases

Spark is used in various industry scenarios:

• ETL and data lakes: pulling data from sources, cleaning, transforming, and writing to Parquet in the data lake — the most common pattern in the world.
• Machine learning: training classification, regression, clustering, and recommendation models with MLlib.
• Real-time analytics: product metrics dashboards with Structured Streaming from Kafka.
• Interactive analytics: answering ad-hoc business questions with Spark SQL.
• Graph processing: network analysis and recommendations with GraphX.

A real example: a fintech company runs nightly ETL of transactions into a data lake using Spark batch, while the product team analyzes user clicks in real time via Structured Streaming. Both use the same engine and API — this is the value of the unified engine Spark offers.

data sources → ingestion → Spark (batch + stream) → warehouse/lakehouse → BI & ML

#Conclusion

Episode 1 gives you the context: Spark was born at the UC Berkeley AMPLab around 2009 to overcome MapReduce's disk-bound weaknesses, became a top-level Apache project in 2014, and is now the most popular distributed engine, with the advantages of in-memory computation and a unified API.

Key takeaways:

• MapReduce writes intermediate results to disk; Spark stores them in memory.
• Spark unifies batch, SQL, streaming, graph, and ML in a single runtime.
• Spark's API is available in Scala, Java, Python, and R with concise syntax.
• Flink excels at true streaming; Beam is an abstraction, not an engine.
• Main use cases: ETL, machine learning, real-time analytics, and interactive SQL.

In the next episode, episode 2, we'll discuss Spark fundamentals and architecture — the roles of the Driver, Executors, and Cluster Manager, the core RDD/DataFrame/Dataset abstractions, the DAG stages and tasks execution model, and the storage model such as partitioning and shuffle. This is the architectural foundation that will accompany the entire series.