Learn Apache Spark - Installing & Running Spark

#Introduction

Episode 2 gave you a map of Spark's architecture. Now it's time to actually run Spark. Episode 3 focuses on practice: standalone installation, interacting through spark-shell and pyspark, submitting jobs with spark-submit, and monitoring execution through the Spark UI.

This skill is the gateway to every following episode. After this episode, you'll be able to run any example code in this series with confidence and understand what's happening behind the scenes.

#Installing Spark Standalone Locally

#Downloading the Binary Release

The fastest way is to download the pre-built binary from the official Apache Spark site. For this series we use version 4.0. Choose a package that already includes Hadoop, then extract:

wget https://archive.apache.org/dist/spark/spark-4.0.0/spark-4.0.0-bin-hadoop3.tgz
tar -xzf spark-4.0.0-bin-hadoop3.tgz
sudo mv spark-4.0.0-bin-hadoop3 /opt/spark

#Setting the Environment Variables

So that Spark commands can be called from anywhere, set SPARK_HOME and PATH. Add to ~/.bashrc:

export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))

After source ~/.bashrc, verify with spark-shell --version. Spark also provides scripts for running a standalone cluster: start-master.sh and start-worker.sh from the sbin directory. We'll use these scripts in episode 14 for deployment.

#Running Spark Shell and PySpark

#Spark Shell for Scala

spark-shell provides an interactive environment in Scala. Inside it, the variables sc (SparkContext) and spark (SparkSession) are already automatically available:

spark-shell --master local[*]
 
scala> val data = sc.parallelize(1 to 10)
scala> data.reduce(_ + _)

The sc.parallelize(1 to 10) command — sorry, the syntax above uses interactive form; we'll cover transformation details in episode 4. The key point: the shell is the fastest experimentation playground because you don't need to compile a full program.

#PySpark for Python

If you chose Python, pyspark provides the same interface:

pyspark --master local[4]

After entering, the spark variable is already active. Try:

First experiment in PySpark

df = spark.range(100).filter("id % 2 == 0")
print(df.count())

Using --master local[4] gives Spark 4 cores so you can see real parallelism. The 50 output shows the filter works correctly.

#Running Jupyter with PySpark

For notebooks, add the findspark dependency or simply install pyspark in the same environment as Jupyter. The spark variable can be created manually with SparkSession.builder as in episode 0 — the most portable pattern across environments.

#Spark Submit and Deployment Modes

#Spark Submit Basics

To run a Spark application stored in a file, use spark-submit. Create a file hitung.py:

Simple hitung.py application

from pyspark.sql import SparkSession
 
spark = SparkSession.builder.appName("hitung").getOrCreate()
df = spark.range(10).filter("id >= 3")
print("Result:", df.count())
spark.stop()

Then submit the job:

spark-submit --master local[2] hitung.py

#Client and Cluster Deployment Modes

In client mode, the driver runs on the machine where spark-submit is executed — easy for debugging because logs appear in the terminal. In cluster mode, the driver runs inside the cluster manager (for example on a worker node), so the submitting machine can be shut down after submission.

spark-submit --master yarn --deploy-mode cluster hitung.py

A practical rule: use client mode during development, cluster mode in production so the driver doesn't depend on a laptop or a CI runner.

#Spark UI and Monitoring

#Opening the Spark UI

While an application is running, Spark provides the Spark UI on port 4040 — you can open it in your browser while the job is active. The UI displays:

• Jobs: the list of jobs and their status.
• Stages: a breakdown of each stage along with duration and task counts.
• Storage: cached data and its size.
• Executors: memory and CPU usage per executor.

open http://localhost:4040

#Understanding the Key Columns

On the Stages page, pay attention to columns like Duration, Shuffle Read/Write, and Tasks. If one stage takes far longer than the others, there's likely skew — some tasks are handling too-heavy data. If Shuffle Read is very large, consider a better join strategy. We'll study both in episode 9.

#Spark History Server

The Spark UI is only available while an application is running. To view the history of finished applications, enable event logging and run the History Server:

spark.eventLog.enabled=true
spark.eventLog.dir=file:/tmp/spark-events

/opt/spark/sbin/start-history-server.sh

With the History Server, you can review the UI of applications that have already finished — very useful for debugging and auditing. Full observability details will be covered in episode 13.

#Conclusion

Episode 3 put the operational foundation into practice: you installed Spark standalone, experimented in spark-shell and pyspark, submitted applications with spark-submit in client and cluster modes, and read the Spark UI and History Server to monitor execution.

Key takeaways:

• Download the pre-built binary and set SPARK_HOME and PATH correctly.
• spark-shell and pyspark are fast playgrounds for experimentation.
• spark-submit uses --deploy-mode client during dev and cluster in production.
• The Spark UI on port 4040 is the main window for viewing jobs, stages, and executors.
• The History Server lets you inspect finished applications.

In the next episode, episode 4, we'll discuss RDD and functional transformations — how to create RDDs, the map, filter, flatMap, reduceByKey, groupByKey operations, the concepts of lazy evaluation and lineage, and when RDDs are still relevant for low-level processing.