Learn Apache Spark - Spark Fundamentals & Architecture

#Introduction

Now that you understand why Spark exists, episode 2 breaks down how Spark works. Understanding Spark's internal architecture is what separates users who merely write code from engineers who can diagnose performance problems.

In this episode we'll unpack four layers: the runtime architecture with the Driver and Executors, the core RDD and DataFrame abstractions, the DAG-based execution model, and the storage model with partitioning and shuffle. This material is the lens you'll use to read every following episode.

#Spark Runtime Architecture

#Driver: The Orchestrator

The Driver is the main process that runs your application code, specifically the main function or driver program. Its tasks: build the DAG (Directed Acyclic Graph), split it into stages and tasks, schedule tasks onto executors, and collect results. The Driver is also where the SparkContext and SparkSession live.

#Executors: The Parallel Workforce

An Executor is a process that runs on worker nodes and executes the tasks scheduled by the driver. Each executor stores cached data and executes code on the partitions of data assigned to it. In general: more executors means more parallelism, but also more cluster memory consumed.

#Cluster Manager: The Resource Provider

The Cluster Manager allocates resources (CPU and memory) for Spark applications. There are three main types: Standalone (Spark's built-in), YARN (from the Hadoop ecosystem), and Kubernetes. The choice of cluster manager doesn't change how you write code, but it determines how resources are shared between applications.

Driver  → builds DAG, schedules tasks, collects results
Executors → execute tasks, store cached data
Cluster Manager → allocates CPU and memory

#Spark Core Abstractions

#RDD: Resilient Distributed Dataset

RDD is Spark's most fundamental abstraction: a collection of elements distributed across the cluster, immutable, and resilient — if a partition is lost, Spark can reconstruct it from lineage. RDD has two kinds of operations: transformations (lazy) and actions (eager). RDDs are rarely used directly nowadays because of their low-level API, but understanding the concept is a must — we'll cover it fully in episode 4.

#DataFrame: Columnar Data with a Schema

A DataFrame is a distributed collection of rows with a schema (typed columns) that leverages the Catalyst optimizer for query optimization. It's the abstraction most often used in PySpark and Spark SQL. Its operations follow the data manipulation patterns of SQL: select, filter, groupBy, join.

Simple DataFrame

from pyspark.sql import SparkSession
 
spark = SparkSession.builder.master("local[*]").appName("abstractions").getOrCreate()
df = spark.createDataFrame([("budi", 25), ("sari", 30)], ["name", "age"])
df.filter(df.age > 26).select("name").show()

df.filter(df.age > 26) shows the declarative pattern: you state what you want, and Spark decides how to run it optimally.

#Dataset: Type-Safe in Scala and Java

Dataset is a DataFrame with a typed API — powerful in Scala and Java because every row is a typed object checked at compile time. In PySpark, Dataset isn't available because Python isn't a static language; we'll cover the concept in episode 6.

#Spark SQL: The SQL Interface

Spark SQL lets you write ordinary SQL queries against data stored in tables or views. This makes Spark usable by analysts who don't write code, and enables code sharing between SQL queries and API programs.

#Execution Model: DAG, Stages, and Tasks

#DAG Scheduler

When you call an action like count() or show(), Spark builds a DAG from all the registered transformations. A DAG is a directed acyclic graph that describes the dependencies between operations. The DAG Scheduler then splits the DAG into stages.

#Stages and Tasks

A stage is a group of tasks that can run together without a shuffle. Stage boundaries appear when a shuffle occurs (for example groupBy or join). Each stage contains many tasks — one task works on one partition of data. These tasks are what get sent to executors for execution.

code → DAG → stages → tasks → schedulers → executors

#Jobs and Scheduling

Each action triggers one job. Within a job, the Task Scheduler sends tasks to the executor closest to the data (data locality). Understanding this flow helps you read the Spark UI when bottlenecks occur — for example, seeing many tasks with unbalanced durations that point to skew.

#Storage Model: Partitioning, Persistence, and Shuffle

#Data Partitioning

Data in Spark is always divided into partitions — logical chunks spread across executors. The number of partitions determines the degree of parallelism. For files on HDFS or S3, one partition defaults to roughly 128MB. You can control the number of partitions with repartition() and coalesce().

#Persistence Levels

Spark can store data in memory and on disk at various levels. Persistence helps avoid recomputation when data is used repeatedly:

MEMORY_ONLY    → store in JVM memory
MEMORY_AND_DISK → memory first, spill to disk when full
DISK_ONLY      → store directly to disk

Using df.cache() or df.persist() requires judgment: storing too much data in memory can cause spills that actually slow things down. We'll optimize caching strategies in episode 9.

#Shuffle: The Main Cost

Shuffle occurs when data needs to be regrouped across executors — such as during groupByKey, join, or reduceByKey. Shuffle requires writing data to disk, network transfer, and re-partitioning. This is the main reason Spark jobs slow down. This entire series will repeatedly stress how to minimize shuffle.

#Conclusion

Episode 2 gives you an architecture map: the Driver builds the DAG and schedules, Executors execute tasks, and the Cluster Manager provides resources. The layered data abstractions from RDD, DataFrame, Dataset, to Spark SQL all run on top of the DAG execution model that splits work into stages and tasks.

Key takeaways:

• The Driver builds the DAG and schedules; executors execute tasks.
• RDD is the foundation; DataFrame uses the Catalyst optimizer.
• Every action triggers a job that splits into stages and tasks.
• Partitions determine parallelism; shuffle is the biggest performance cost.
• Persistence helps, but used wrongly it actually slows things down.

In the next episode, episode 3, we'll discuss installing and running Spark — downloading the binaries, running spark-shell and pyspark, using spark-submit with client and cluster deployment modes, and reading the Spark UI on port 4040. All these architecture principles will soon become visible in practice.