Learn Apache Spark - Data Sources & Storage

#Introduction

So far you've mastered transformations, joins, and aggregations. But all those operations mean nothing without data. Episode 8 covers the lowest yet most decisive layer of a data pipeline: how Spark connects to storage, which format is best for a given workload, and how to organize files so reads stay fast as data grows.

Storage and format choices directly impact cost and speed. Reading 1 TB of CSV data can take many times longer than reading 1 TB of Parquet data, even with the same hardware. That's why understanding data sources and storage isn't just extra knowledge — it's an architectural decision that affects the entire pipeline.

In this episode we'll cover four things: connecting Spark to HDFS, S3, and JDBC; the file formats Parquet, Avro, ORC, JSON, and CSV; partitioning and bucketing strategies; and schema evolution and compatibility.

#Connecting Spark to Various Storage Systems

#HDFS

HDFS (Hadoop Distributed File System) is the classic distributed storage in the Hadoop ecosystem. Spark reads HDFS through the hdfs:// URI scheme, and file locations are written as Hadoop paths:

hdfs://namenode:8020/data/penjualan.parquet

To use HDFS, the core-site.xml and hdfs-site.xml configuration files must be available on the Spark classpath, and fs.defaultFS must point to the correct NameNode. The main advantage of HDFS is data locality — Spark can schedule tasks on the node where the data resides, reducing network transfer.

#Object Storage S3 and GCS

In the cloud era, object storage is more common than HDFS. For S3, Spark uses the S3A connector with credentials set through environment variables or Hadoop configuration:

export AWS_ACCESS_KEY_ID=xxxx
export AWS_SECRET_ACCESS_KEY=yyyy
export AWS_REGION=ap-southeast-1

After that, files on S3 can be read like ordinary paths: spark.read.parquet("s3a://bucket/data/"). For GCS, use the gcs-connector connector and the gs:// path. Object storage doesn't have the data locality of HDFS, but it offers unlimited capacity and far simpler operational costs.

#JDBC Databases

Spark can also read and write directly to relational databases via JDBC. This is important for integration with warehouses or transactional applications:

Reading a table via JDBC

df = spark.read \
    .format("jdbc") \
    .option("url", "jdbc:postgresql://host:5432/db") \
    .option("dbtable", "penjualan") \
    .option("user", "spark") \
    .option("password", "rahasia") \
    .option("numPartitions", 8) \
    .load()

The numPartitions parameter determines how many parallel tasks read from the database. For large tables, add partitionColumn, lowerBound, and upperBound so the reads are evenly split — otherwise the database can be overwhelmed by one giant query.

#File Formats: Reading and Writing

#Parquet: The De Facto Standard

Parquet is a compressed columnar format designed for analytics. Because each column's data is stored contiguously, queries that select few columns only read the data they need. Parquet stores the schema inside the file, so no additional configuration is needed:

Writing and reading Parquet

df.write.format("parquet").mode("overwrite").save("data/penjualan.parquet")
kembali = spark.read.format("parquet").load("data/penjualan.parquet")

df.write.mode("overwrite") replaces old data — switch to "append" if you want to add new data. Parquet is the primary choice in almost every data lake because of its small file size and high read speed.

#Avro and ORC

Avro is a row-based format that stores the schema alongside the data, popular for streaming and message queues because it's efficient for record-by-record serialization. ORC is a columnar format from the Hive ecosystem that is also highly compressed. Both are read with the same API:

Reading Avro and ORC

avro_df = spark.read.format("avro").load("data/event.avro")
orc_df = spark.read.format("orc").load("data/warehouse.orc")

Avro excels for pipelines that write records one at a time; Parquet and ORC excel for analytics that read many columns at once.

#JSON and CSV

CSV is the simplest tabular format but also the most wasteful: no schema, no columnar compression, and slow parsing. JSON is suited to semi-structured data and logs:

Reading JSON and CSV

json_df = spark.read.format("json").option("multiLine", True).load("data/event.json")
csv_df = spark.read.format("csv").option("header", True).load("data/transaksi.csv")

Use CSV and JSON for ingesting from external sources, then convert them to Parquet promptly for primary storage. Keeping raw data forever in row-based formats is one of the most common reasons pipelines become expensive.

#Partitioning, Bucketing, and File Layout

#Partitioning

Partitioning splits data into folders based on column values:

Writing with partitioning

df.write \
    .format("parquet") \
    .partitionBy("tahun", "bulan") \
    .save("data/penjualan")

The result is a folder structure like data/penjualan/tahun=2026/bulan=01/. When a query filters on a specific year and month, Spark only reads the relevant folders — this is called partition pruning. Choose partition columns that are balanced: too many unique values makes folders too small and actually slows things down.

#Bucketing

Bucketing splits data into a fixed number of files based on a column hash:

Writing with bucketing

df.write \
    .format("parquet") \
    .bucketBy(16, "kota_id") \
    .sortBy("tanggal") \
    .saveAsTable("penjualan_bucketed")

If both sides of a join are bucketed on the same column with the same bucket count, Spark can join without a full shuffle because matching data already sits in the same bucket. This is one of the most impactful layout optimizations.

#File Layout

General file layout rules: files around 128MB to 256MB, not too many files but enough for parallelism, and avoid thousands of tiny files that overwhelm the NameNode and driver. Techniques like coalesce() and repartition() before writing help control the number of files.

#Schema Evolution and Compatibility

#Why Schemas Change

Data is always changing: new columns get added, types change, or fields are removed. Schema evolution is the ability of a system to read old data whose schema differs from new data. Parquet and Avro provide clear evolution rules: adding columns with default values, reducing type precision, and so on.

adding a nullable column       → safe
adding a non-nullable column   → a default is required
narrowing a type               → risky, avoid
removing a column              → old data remains readable

#Applying a New Schema

When a schema changes, use these strategies:

• Spark: read with a merged schema via spark.sql.mergeSchema for Parquet.
• Delta Lake or Iceberg: manage evolution explicitly with ALTER TABLE — covered in episode 15.
• Versioning: store data in versioned folders or a schema_version column if it changes often.

The key to schema evolution is discipline: define the schema in one place, test backward compatibility, and don't let file formats grow unchecked on their own.

#Conclusion

Episode 8 equips you with the storage foundation: Spark reads from HDFS, S3, GCS, JDBC, and the file system; the Parquet, Avro, ORC, JSON, and CSV formats each have their own trade-offs; partitioning and bucketing control how files are organized; and schema evolution must be planned, not left to chance.

Key takeaways:

• Parquet is the primary format for analytics; JSON and CSV are only for ingestion.
• HDFS offers data locality, object storage offers scalability.
• partitionBy enables partition pruning when queries filter on it.
• Bucketing with equal bucket counts on both join sides avoids large shuffles.
• Schema evolution requires disciplined compatibility testing.

In the next episode, episode 9, we'll discuss performance tuning and optimization — memory and shuffle configuration, broadcast joins, caching and persistence, reading the Catalyst execution plan, and strategies to avoid data skew and shuffle overload, which are the main causes of slow production queries.