Learn Apache Spark - RDD & Functional Transformations

#Introduction

All of Spark's high-level abstractions — DataFrame, Dataset, Spark SQL — ultimately run on top of a foundation called RDD. Understanding RDD means understanding the machine under the hood: how data is distributed, how transformations stack up, and how Spark survives failures.

Episode 4 covers RDD thoroughly: how to create one, the two types of operations (transformations and actions), the most important functional functions, lazy evaluation, lineage, and when RDD is still the right choice in the age of DataFrames.

#RDD Basics

#What Is an RDD

An RDD (Resilient Distributed Dataset) is a collection of elements that is immutable, distributed across the cluster, and can be processed in parallel. The word resilient means that if some data is lost because an executor fails, Spark can rebuild that partition from the computation trail (lineage) without having to redo the entire job.

An RDD stores any kind of raw data: text, numbers, Scala objects, or structural records. Each RDD is divided into partitions spread across executors — this is the unit of parallelism that determines how many tasks run at the same time.

#Creating an RDD

There are several ways to create an RDD. The most common in experiments is parallelize, and in the real world it's reading from a storage source:

scala> val rdd = sc.parallelize(Seq("budi", "sari", "dewi"))
scala> rdd.partitions.size

In PySpark, the equivalent is sc.parallelize(["budi", "sari", "dewi"]). For file data, sc.textFile("data/transactions.csv") creates one RDD of strings per line. The number of partitions can be set via the second argument of parallelize.

#Transformations and Actions

The key concept: RDD operations are divided into two types.

• Transformation: a lazy operation that produces a new RDD, e.g. map, filter, flatMap. No computation runs yet.
• Action: an operation that triggers execution and returns a value, e.g. count(), collect(), reduce().

scala> val rdd2 = rdd.map(_.toUpperCase)   // transformation, lazy
scala> rdd2.collect()                      // action, triggers execution

Until an action occurs, Spark only builds a pipeline of operations that hasn't been executed. This is what allows Spark to optimize the overall execution order.

#Main Functional Functions

#Map, Filter, and FlatMap

The three most basic transformations:

map, filter, flatMap in PySpark

sc = spark.sparkContext
angka = sc.parallelize([1, 2, 3, 4, 5])
 
hasil_map = angka.map(lambda x: x * 2)                # [2, 4, 6, 8, 10]
hasil_filter = angka.filter(lambda x: x % 2 == 0)     # [2, 4]
hasil_flatmap = angka.flatMap(lambda x: [x, x + 1])   # [1,2,2,3,3,4,...]

• map transforms each element into exactly one new element.
• filter keeps the elements that satisfy a predicate.
• flatMap transforms one element into zero or more elements, then flattens the results — ideal for splitting a row into words.

Classic word count with flatMap

kalimat = sc.parallelize(["halo dunia", "halo spark"])
kata = kalimat.flatMap(lambda s: s.split(" "))
print(kata.collect())

The result ['halo', 'dunia', 'halo', 'spark'] shows how flatMap splits and flattens — the foundation of the legendary word count algorithm.

#ReduceByKey and GroupByKey

For paired data (key, value), Spark provides key-based aggregation:

reduceByKey vs groupByKey

pasangan = sc.parallelize([("apel", 3), ("pisang", 2), ("apel", 1)])
 
by_key = pasangan.reduceByKey(lambda a, b: a + b)   # [('apel',4),('pisang',2)]
grouped = pasangan.groupByKey().mapValues(list)     # [('apel',[3,1]),...]

An important difference: reduceByKey combines values before the shuffle, so far less data is sent between nodes. groupByKey sends all raw values to the same partition — often slower for simple aggregations. A good rule of thumb: choose reduceByKey whenever possible.

#Lazy Evaluation and Lineage

#Lazy Evaluation

Spark delays execution until an action occurs. This isn't just a technique — it's what enables optimization: Spark can combine chained transformations, prune unused columns, and avoid wasted work. If you build a hundred transformations without an action, not a single one is executed.

#Lineage Graph

Each RDD stores the trail of transformations that formed it — this is its lineage. If a partition is lost, Spark traces the lineage and recomputes only that partition. This is a fault tolerance mechanism that requires no data replication, an architectural advantage of RDD from the very beginning.

textFile → flatMap → map → filter → reduceByKey

You can display an RDD's lineage with rdd.toDebugString — very useful when you want to understand where an RDD came from or why a recompute is happening.

#When RDD Is Still Relevant

#The Strengths of RDD

RDD is still relevant when you need the lowest-level control: manipulating data that isn't tabular, building custom transformation libraries, or working with data that has no fixed schema. Much of the classic MLlib ecosystem and several third-party libraries still consume RDDs.

#When to Leave RDD Behind

For most workloads, DataFrame is the better choice:

• Automatic optimization: the Catalyst optimizer prunes and filters as early as possible.
• Columns and data types: the schema avoids hidden bugs.
• Ecosystem: DataFrame integrates with Spark SQL, streaming, and MLlib.

A DataFrame is essentially an RDD with a schema and an optimizer — so learning RDD in this episode is still valuable, because all the concepts of partitions, transformations, and actions apply the same way at the layers above.

columnar & structured data → DataFrame / Dataset / SQL
low-level control & free-form data → RDD

#Conclusion

Episode 4 rounds out your understanding of the foundation: RDD is a distributed, immutable, and resilient collection, executed through lazy transformations and eager actions. Functional operations like map, filter, flatMap, reduceByKey, and groupByKey are the basic vocabulary that recurs throughout every layer of Spark.

Key takeaways:

• RDD is the distributed foundation; DataFrame is an RDD with a schema.
• Transformations are lazy, actions are eager — no computation without an action.
• flatMap splits and flattens; reduceByKey is more efficient than groupByKey.
• Lineage allows recomputing lost partitions without replication.
• Choose RDD only for low-level control; use DataFrame for most cases.

In the next episode, episode 5, we'll discuss DataFrame and Spark SQL — creating DataFrames from CSV, JSON, and Parquet, running SQL with temporary views, understanding schema inference and the Catalyst optimizer, and the transformations and expression API that become Spark's everyday language.