Learn Apache Spark - Resource Management & Deployment

#Introduction

A job that runs on a laptop isn't enough for production. Episode 14 covers how Spark is deployed to a real cluster and how resources are allocated efficiently among many applications. This is the bridge from development to operations.

The choice of cluster manager determines how resources are shared, how jobs are scheduled, and how failure recovery happens. Meanwhile, executor configuration determines how much parallelism you get — and how big the cloud bill is. Both must be managed deliberately.

This episode covers four topics: the deployment options standalone, YARN, Mesos, and Kubernetes; the Spark on Kubernetes pattern; managing executors, cores, and memory; and dynamic allocation and workload isolation.

#Deployment Options: Standalone, YARN, Mesos, and Kubernetes

#Standalone

Standalone is Spark's built-in cluster manager. It's simple and easy to debug — suited to development and small clusters. Start a master and worker:

/opt/spark/sbin/start-master.sh
/opt/spark/sbin/start-worker.sh spark://localhost:7077

Then submit with spark-submit --master spark://localhost:7077. Its downsides: no strict multi-tenant isolation and limited security features.

#YARN and Mesos

YARN is the Hadoop ecosystem cluster manager that shares resources based on containers. It fits if your infrastructure already uses HDFS and the Hadoop ecosystem. Mesos supports mixed workloads, but its adoption keeps declining and it's rarely chosen for new projects.

spark-submit --master yarn --deploy-mode cluster job.py

#Kubernetes

Kubernetes has become the primary choice for modern projects because of its mature resource provisioning, scaling, and ecosystem. Spark 3+ supports Kubernetes as a native cluster manager, where the driver and executors run as pods.

#Spark on Kubernetes: Patterns and Pod Management

#How It Works

When you submit to Kubernetes, Spark creates a driver pod in a given namespace, and the driver then requests executor pods as needed:

spark-submit \
  --master k8s://https://k8s-api.example.com \
  --deploy-mode cluster \
  --conf spark.kubernetes.container.image=registry/app/spark:4.0.0 \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa \
  job.py

--master k8s://... points Spark at the Kubernetes API. The Spark image must already be built — usually from a base image that adds your application libraries and required credentials.

#Recommended Patterns

Several patterns worth noting:

• Pod template: customize pods with tolerations, node selectors, or volumes via spark.kubernetes.driver.podTemplateFile.
• A dedicated ServiceAccount for Spark so you don't grant excessive access.
• Dynamic allocation with executors created and removed as needed (see the final section).
• RBAC: limit the APIs Spark pods can access via Roles and RoleBindings.

driver pod → request executor pods → executors run tasks → results return

#Managing Executors, Cores, and Memory

#Sizing Executors

Two practical formulas when setting up executors:

spark-submit \
  --executor-memory 8g \
  --executor-cores 4 \
  --num-executors 20 \
  job.py

A general rule: reserve about 10-15 percent of JVM overhead above --executor-memory, and cap --executor-cores so you don't monopolize a node. For example, on a node with 64GB RAM and 16 cores, eight executors with 8GB and 2 cores each is a reasonable split.

#Avoiding Imbalance

• Don't go too small: many small executors means lots of scheduling overhead.
• Don't go too large: giant executors make co-locating other applications harder and increase the risk of failure.
• Match shuffle partitions to total cores: spark.sql.shuffle.partitions around 2-3 times the total cores is a good starting point.

#Dynamic Allocation and Workload Isolation

#Dynamic Allocation

With dynamic allocation, Spark adds and removes executors automatically based on load:

spark.dynamicAllocation.enabled      true
spark.dynamicAllocation.shuffleTracking.enabled true
spark.dynamicAllocation.initialExecutors 2
spark.dynamicAllocation.maxExecutors 40

spark.dynamicAllocation.maxExecutors caps the maximum executors so the whole cluster isn't consumed. This feature is very useful in multi-tenant environments: idle jobs release executors, and busy ones get them back when load rises.

#Workload Isolation

To prevent one job from wasting shared resources:

• Executor limits: set an explicit upper bound; don't leave everything to the scheduler.
• Separate namespaces/queues: split batch and streaming jobs into different YARN queues or Kubernetes namespaces.
• Priority: use scheduling pools (e.g. spark.scheduler.pool) so important jobs are served first.
• Kubernetes ResourceQuota: cap total CPU and memory per Spark namespace.

batch queue (40%)  → daily ETL, bounded concurrency
streaming queue (30%) → real-time pipelines, high priority
ad-hoc queue (30%) → analyst experiments, low max executor limit

#Conclusion

Episode 14 takes you into the operational domain: Standalone for simple cases, YARN for the Hadoop ecosystem, Kubernetes for modern, cloud-native setups. You also understand how to size executors, leverage dynamic allocation, and maintain workload isolation so one job doesn't disturb another.

Key takeaways:

• Standalone for development, YARN for the Hadoop ecosystem, Kubernetes for cloud-native.
• Spark on Kubernetes runs the driver and executors as pods.
• Executor sizing must balance memory, cores, and JVM overhead.
• Dynamic allocation adjusts executors to actual load.
• Workload isolation uses queues, namespaces, and explicit executor limits.

In the next episode, episode 15, we'll discuss advanced SQL and analytics — CTEs, subqueries, and nested queries, UDFs and UDAFs, spatial and graph analytics with advanced windowing, and integration with Delta Lake and Iceberg.