Learn Apache Spark - Dataset API & Strong Typing

#Introduction

DataFrame offers convenience, but sometimes you need type guarantees at compile time — not at runtime. That's where Dataset comes in: a typed abstraction that's only fully available in Scala and Java.

Episode 6 covers Dataset in depth: what a Dataset is, how encoders provide type safety, its structural differences from DataFrame, and how to move between the two. This material is most relevant if you write Spark in Scala or Java, and it remains valuable as architectural knowledge for PySpark users.

#Using Dataset in Scala and Java

#The Basic Dataset Concept

A Dataset is a distributed collection of typed objects. In Scala, a Dataset[T] stores rows as objects of type T — which can be a case class, a tuple, or a primitive type. In Java, the type is Dataset<T> with JavaBeans or Encoders for specific types.

Dataset from a case class in Scala

import org.apache.spark.sql.SparkSession
 
case class Pegawai(nama: String, gaji: Double)
 
val spark = SparkSession.builder.master("local[*]").appName("dataset").getOrCreate()
import spark.implicits._
 
val ds: Dataset[Pegawai] = Seq(
  Pegawai("budi", 8000000),
  Pegawai("sari", 9500000)
).toDS()
 
ds.filter(_.gaji > 8500000).show()

Note ds.filter(_.gaji > 8500000) — here _ refers to the whole Pegawai object, not a Row. The compiler checks that gaji is a valid field of the case class, so field name errors are caught at compile time instead of in the middle of the cluster.

#Dataset in Java

In Java, Dataset works with classes that follow the JavaBeans convention:

Dataset<Pegawai> ds = spark.read().csv("data/pegawai.csv")
    .as(Encoders.bean(Pegawai.class));

Encoders.bean(Pegawai.class) tells Spark how to convert data into Pegawai objects. The syntax is more verbose than Scala's, but the same type safety guarantees still apply.

#Encoders and Compile-Time Type Safety

#What Is an Encoder

An Encoder is the mechanism that converts JVM objects into Spark's internal representation and back. Every data type has an encoder: case classes get an automatic encoder via spark.implicits, while special types need explicit Encoders.*.

Explicit encoder for a special type

import org.apache.spark.sql.Encoders
 
val ds: Dataset[Pegawai] = spark
  .read
  .option("header", "true")
  .csv("data/pegawai.csv")
  .as(Encoders.product[Pegawai])

With Encoders.product[Pegawai], Spark validates that the columns in the file match the case class fields. Type mismatches such as a string column versus a Double field will report a clear error.

#The Benefits of Type Safety

The real benefits of type safety:

• Earlier errors: field or type mistakes are caught at compile time.
• Safe refactoring: renaming a case class field immediately flags all incorrect usages.
• Readable code: pipelines look like ordinary Scala collection manipulation.

#Dataset Structure vs DataFrame

#The Relationship Between Them

Architecturally, DataFrame is an alias for Dataset[Row]. In the Spark Scala API, DataFrame is actually a type alias — there's no separate class. This explains why all DataFrame operations are available to Dataset and vice versa.

DataFrame = Dataset[Row]   (rows typed as Row, no specific type)
Dataset[T] = rows typed as object T (type-safe)

#When to Choose Which

• DataFrame: untyped, suitable for data without an object model, SQL integration, and quick exploration. Its API uses string column names.
• Dataset: typed, suitable for pipelines that need long-term maintainability, domain objects, and compile-time checks.

There's no single right answer. Many teams use DataFrame at the ingestion and exploration layers, then switch to Dataset for the core business transformation layer.

#Performance Trade-offs

Both APIs compile to the same physical plan after passing through Catalyst, so the performance difference is generally small. However, Row-oriented operations (like map and flatMap on Dataset) can be slower than declarative operations because they involve object serialization. For simple transformations, prefer Spark expressions over Scala map when possible.

#Converting Between DataFrame and Dataset

#From DataFrame to Dataset

The conversion is done with .as[T]:

Converting DataFrame to Dataset

val df = spark.read.option("header", "true").csv("data/pegawai.csv")
val ds: Dataset[Pegawai] = df.as[Pegawai]

df.as[Pegawai] validates that the column schema matches the case class fields. If any column doesn't match, an error appears immediately.

#From Dataset to DataFrame

Conversely, just call toDF():

Converting Dataset to DataFrame

val dfKembali = ds.toDF()

ds.toDF() converts typed objects back into a Dataset[Row]. This is useful when you want to write results with SQL or store them in a table that requires a Row representation.

Dataset[T] .as[T] from DataFrame   ← df.as[Pegawai]
DataFrame  toDF from Dataset       ← ds.toDF()

#Conclusion

Episode 6 opens up Spark's typed layer: Dataset provides compile-time type safety through encoders, is fully available in Scala and Java, and architecturally a DataFrame is just Dataset[Row]. The two-way conversion lets you use both APIs side by side as needed.

Key takeaways:

• Dataset is a distributed collection of typed T objects.
• DataFrame is an alias for Dataset[Row] — not a separate class.
• Encoders handle JVM object conversion and provide type safety.
• Type errors are caught at compile time, not in the middle of the cluster.
• Conversion via .as[T] and .toDF() is two-way and cheap.

In the next episode, episode 7, we'll discuss joins, aggregations, and window functions — every join type from inner to anti, groupBy aggregations, window functions like row_number and lead, and the performance implications of joins and shuffle that are the main causes of slow queries.