Learn Apache Spark - Security & Authentication

#Introduction

Spark clusters process data that is often sensitive: customer transactions, user logs, even financial data. Episode 12 covers the layer that beginners often overlook but production demands: security. Without proper protection, a single exposed executor could leak an entire dataset.

Spark security isn't a single feature but a set of layers: encrypting traffic between components, authenticating who is allowed to run jobs, authorizing what can be accessed, and managing credentials so they don't leak into code or logs.

This episode covers four areas: TLS/SSL configuration for the cluster, authentication and authorization for jobs, securing data sources and credentials, and integration with Kerberos and LDAP.

#Securing the Cluster with TLS/SSL

#Why TLS Is Necessary

Without encryption, traffic between the driver, executors, and cluster manager flows in plain text — readable by anyone who taps the network. TLS/SSL encrypts communication between Spark components. In Spark 4, this is controlled by parameters in the spark.ssl namespace.

#Enabling Component Encryption

For a standalone cluster, set the following configuration in spark-defaults.conf:

spark.ssl.enabled                   true
spark.ssl.protocol                  TLSv1.3
spark.ssl.standalone.enabled        true
spark.ssl.standalone.trustStore     /etc/spark/truststore.jks
spark.ssl.standalone.trustStorePassword  changeit
spark.ssl.standalone.keyStore       /etc/spark/keystore.jks
spark.ssl.standalone.keyStorePassword    changeit

spark.ssl.standalone.enabled enables encryption of communication between the driver, master, and workers. Each process needs a keystore containing its private certificate and a truststore containing the trusted CAs. Make sure the passwords are stored in a secret manager, not in a config that goes into the repository.

#Encryption Inside the SparkContext

In addition to inter-component communication, also enable encryption for shuffle data and RDD block transfer that runs between executors:

spark.network.crypto.enabled        true
spark.authenticate.enableSaslEncryption  true
spark.io.encryption.enabled         true

spark.network.crypto.enabled uses SASL for authentication and encryption of data transfer between nodes. The combination of the three closes the data leak gap mid-flight between executors.

#Authentication and Authorization for Spark Jobs

#Authentication with a Shared Secret

To prevent just anyone from submitting jobs to the cluster, standalone Spark supports shared secret-based authentication set on the master and workers:

export SPARK_AUTHENTICATE=true
export SPARK_AUTHENTICATE_SECRET=rahasia-cluster

All nodes must use the same secret, and the secret should ideally be distributed through a mechanism like a Kubernetes Secret, not hardcoded in scripts. Applications that try to submit without the correct secret are rejected.

#Access-Level Authorization

For finer-grained control, install Apache Ranger or integrate with the Ranger Spark Plugin, which provides user-based policies on tables and columns. Another alternative is installing Shiro (spark.authenticate and spark.ui.acls) to control who can view the Spark UI and job history.

shared secret    → who may submit jobs
Ranger/ACLs      → who may access tables, columns, and the UI
Kerberos/LDAP    → centralized user identity

#Securing Data Sources and Credentials

#Don't Hardcode Credentials

Credentials written directly in code are one of the biggest sources of leaks. Always pull them from environment variables or a secret manager:

Credentials from the environment

import os
 
df = spark.read \
    .format("jdbc") \
    .option("url", os.environ["JDBC_URL"]) \
    .option("user", os.environ["JDBC_USER"]) \
    .option("password", os.environ["JDBC_PASSWORD"]) \
    .load()

Reading os.environ["JDBC_PASSWORD"] ensures credentials never end up in code files or the repository. On Kubernetes these values are injected via Kubernetes Secrets; on Databricks or EMR, via their respective secret managers.

#Protection from Logs

Some connectors print configuration during debugging — credentials can leak into logs. Important practices:

• Use passwords via spark-submit --properties-file, with that file given strict permissions.
• Enable the right logging levels so configuration isn't printed.
• Redact sensitive values in the monitoring pipeline.

#Integration with Kerberos and LDAP

#Kerberos for the Hadoop Ecosystem

In environments with HDFS and YARN, user identity is managed with Kerberos. Spark uses credential delegation tokens obtained from the KDC (Key Distribution Center) to access HDFS and other services.

kinit -kt /etc/security/spark.keytab spark@REALM.LOKAL
spark-submit --master yarn --keytab /etc/security/spark.keytab \
  --principal spark@REALM.LOKAL job.py

--keytab and --principal tell Spark what identity to use when interacting with HDFS and YARN. Make sure the keytab is rotated periodically and only readable by its owner.

#LDAP for Centralized Identity

LDAP/Active Directory provides a centralized user directory. A common integration: users are authenticated via LDAP at the gateway (for example through Livy or Kubernetes), and data authorization is determined by group attributes in the same directory.

user → LDAP (authentication) → Kerberos ticket → HDFS access → Ranger policy

This chain makes a single identity flow from login to data access, so the audit trail can be tracked per user.

#Conclusion

Episode 12 rounds out the security foundation: TLS/SSL encrypts inter-component communication and shuffle data, shared secrets and policies control who may submit and access, credentials are kept out of code and logs, and Kerberos and LDAP provide centralized identity for enterprise environments.

Key takeaways:

• Enable TLS for master, workers, driver, and executors communication.
• Encrypt shuffle data with network crypto and io encryption.
• Don't hardcode credentials; read them from the environment or a secret manager.
• Kerberos is used for HDFS and YARN access in the Hadoop ecosystem.
• LDAP and Ranger provide centralized identity and access policies.

In the next episode, episode 13, we'll discuss observability and monitoring — collecting metrics and logs, integrating Prometheus and Grafana, leveraging the History Server and event logs, and setting up alerting for failed jobs and data skew.