Learn Apache Spark - Pre-Requisite Skills & Environment Setup

#Introduction

Welcome to the Learn Apache Spark series! This series will take you from the foundational concepts all the way to production readiness with Apache Spark — the most widely used distributed data processing engine in the data engineering world. There are 23 episodes in total, organized into six phases.

But before you run your first spark-submit, there are some core skills and software you must have in place. Why are these prerequisites important? Because Spark isn't just an ordinary Python library. It's a distributed engine that runs work across many machines, so an understanding of the JVM, clusters, and parallel processing is a non-negotiable foundation.

Episode 0 is your roadmap: we'll make sure your core skills are solid, set up the Java JDK, install Apache Spark, verify your first installation, and prepare additional tools like Docker and Git. Once this episode is done, you can comfortably follow the entire series.

#Core Skills You Must Master

#Data Engineering Fundamentals and Big Data Concepts

You need to understand basic data engineering concepts: what ETL (extract, transform, load) is, batch processing versus stream processing, and how data flows from sources to analytical storage. Also understand the big data dimensions of volume, velocity, and variety — because together they determine why a single machine isn't enough and why we need a cluster.

#Familiarity with JVM, Python, or Scala

Spark is a JVM application. Spark's core code is written in Scala and runs on top of the Java Virtual Machine. You just need to be comfortable with one of: Python (for PySpark), Scala (for Spark's native API), or Java. Python is the most popular entry point because of its concise syntax, while Scala gives you the fullest control over Spark's API.

java -version
python3 --version

Make sure Java shows version 11 or 17, and Python shows version 3.9 or newer. We'll use both throughout the series.

#Distributed Computing and Cluster Architecture

The most important concept: Spark splits data into partitions, sends tasks to many executors running in parallel, then combines the results. Understand the terms master, worker, node, and network shuffle — the exchange of data between nodes that is the main source of performance cost. We'll dig deep into this architecture in episode 2.

#Basic Understanding of SQL and Data Formats

Spark SQL uses SQL as one of its main interfaces. You must be comfortable with SELECT, WHERE, GROUP BY, and JOIN. Also get to know the common data formats: CSV, JSON, Parquet, and Avro. Parquet is the columnar format most often used in the Spark world because of its compression and read speed — we'll cover it in episode 8.

#Software You Need to Prepare

#Java JDK 11 or 17

The first step is making sure the JVM is available. On Ubuntu, install OpenJDK 17:

sudo apt update
sudo apt install openjdk-17-jdk
java -version

The latest version of Spark requires Java 17, so JDK 17 is the primary recommendation. If your system already has Java 11, that's still supported, but it's better to follow the latest standard.

#Downloading and Setting Up Apache Spark

Download the Spark binary release from the official apache.org site. For this series, use the stable Spark 4.0 version. Extract it to a directory like /opt/spark and set the environment variables:

export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64

To make this permanent, add the lines above to your ~/.bashrc file. Verify:

spark-shell --version

The spark-shell --version command displays the installed Spark and Scala versions. If it shows version 4.0.x, your installation is correct.

#Storage Layer: HDFS or Object Storage

For episodes 8-9 later, you'll need storage for your data. There are two options: HDFS (the distributed file system from the Hadoop ecosystem) or object storage like S3 and GCS. For local learning, you can simply use the regular file system, because Spark treats local folders as a data source. However, setting up Docker for HDFS will be very helpful in phase 3.

#Python Environment with PySpark and Additional Tools

If you choose PySpark, create a virtual environment and install pyspark:

python3 -m venv .venv
source .venv/bin/activate
pip install pyspark

The pyspark package ships with the Spark binaries, so you don't need to download them manually if you're only using Python. Recommended tools as well: Docker for running HDFS or a local cluster, Git for version control, and an editor such as VS Code or Jupyter Notebook.

#Hardware Minimum Requirements

Spark demands memory because it runs as a JVM process. The minimum recommendation for following this series: 8GB RAM (16GB is more comfortable), a quad-core CPU, and 20GB+ SSD for storing data and the Spark application. Modern laptops generally meet these requirements.

#Environment Verification

Before moving on to episode 1, run a full verification. Make sure all of the following commands produce output without errors:

java -version        → Java 17
python3 --version    → Python 3.9+
spark-shell --version → Spark 4.0.x
docker --version     → Docker Engine active
git --version        → Git available

If all the lines above run, your environment is ready. Also do a quick PySpark test with the following code:

First PySpark test

from pyspark.sql import SparkSession
 
spark = SparkSession.builder.master("local[*]").appName("check").getOrCreate()
df = spark.range(5)
print(df.collect())
spark.stop()

The SparkSession.builder.master("local[*]") call creates a local session that uses all CPU cores — this is the most common pattern for development. If df.collect() returns [Row(id=0), ..., Row(id=4)], then the whole stack is ready to use.

#Summary of Skills You Must Master

Here's a summary of the prerequisites you've prepared in episode 0:

• Java JDK 17 installed with the correct JAVA_HOME.
• Apache Spark 4.0 downloaded and spark-shell --version working.
• Python 3.9+ with a venv and the pyspark package.
• Docker and Git for the upcoming phases.
• Hardware with at least 8GB RAM and a 20GB SSD.

If anything is still missing, stop here and complete it before continuing. A solid environment foundation will make the next 22 episodes run much more smoothly.

#Conclusion

In episode 0 you've laid the foundation for the entire series: understanding the core data engineering and distributed computing skills, installing the Java JDK 17, setting up Apache Spark 4.0, creating a PySpark environment, and verifying your first operations.

Key takeaways:

• Spark is a JVM engine: master Java 17 and understand the role of the JVM.
• Master the concepts of partitions, executors, and network shuffle from the start.
• Install Spark 4.0 and make sure spark-shell --version works.
• Use a venv for PySpark so dependencies don't clash.
• Always verify your environment before starting to learn new material.

In the next episode, episode 1, we'll discuss the history, background, and why choose Spark — the evolution from MapReduce and the Hadoop ecosystem, the advantages of in-memory computation, its differences from Hadoop MapReduce, Flink, and Beam, and real-world use cases in industry. Make sure your environment is ready, because the Learn Apache Spark journey is just beginning!