Learn Apache Spark - Observability & Monitoring

#Introduction

Spark jobs can run for hours. Without good observability, failures are only detected when it's already too late — data doesn't arrive, quality drops, or the cluster sits idle unnoticed. Episode 13 covers observability: the ability to see what's happening inside the cluster in real time and comprehensively.

Observability is more than reading logs. It includes measurable metrics, readable dashboards, and alerts that arrive before a problem becomes serious. Engineers who master observability can answer the question "why is this job slow?" in minutes, not hours.

This episode covers four pillars: metrics, logs, and the Spark web UI; Prometheus and Grafana integration; the History Server with event logs; and alerting for failed jobs and skew.

#Metrics, Logs, and the Spark Web UI

#The Spark Web UI as the Starting Point

The Spark UI on port 4040 is the first window into observability. Its pages show real-time status:

• Jobs: the list of jobs with their duration and status.
• Stages: stage details, task counts, and shuffle read/write sizes.
• Storage: cached data and used memory.
• Executors: memory, disk, and CPU usage per executor.

open http://localhost:4040

From the Executors page, you can see if one executor is using far more memory than the others — an early indication of data skew. This is the first triage step before looking at other metrics.

#Driver and Executor Logs

Logs live in two places: driver logs appear in the terminal where spark-submit was run (client mode), and executor logs are scattered across worker nodes. In cluster mode, direct logs to a central location so they're easy to search:

spark.eventLog.enabled         true
spark.eventLog.dir             file:/var/log/spark/events
spark.history.fs.logDirectory  file:/var/log/spark/events

spark.eventLog.enabled writes event logs to the directory — this is the fuel for the History Server and also a data source for log aggregation.

#Prometheus and Grafana Integration

#Exporting Metrics to Prometheus

Spark provides a Prometheus-format metrics endpoint through the sparkPrometheus module. Enable it in metrics.properties:

*.sink.prometheusServlet.class   org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path    /metrics/prometheus
*.sink.prometheusServlet.period  10

Once enabled, metrics are available at http://driver:4040/metrics/prometheus. For cluster mode applications whose driver moves, use spark.metrics.executorMetricsSource.enabled=true and let Prometheus do service discovery on YARN or Kubernetes.

#Prometheus and Grafana Configuration

Point Prometheus to scrape the driver and executors every few seconds:

scrape_configs:
  - job_name: spark
    static_configs:
      - targets: ["driver-host:4040", "worker-1:4040", "worker-2:4040"]
        labels:
          cluster: produksi

With Prometheus data, build Grafana dashboards showing key metrics: shuffle read bytes, task completion time, memory used per executor, and active jobs. These dashboards become the data team's daily monitoring board.

#History Server, Event Logs, and Performance Dashboards

#Running the History Server

The event logs enabled earlier can be re-read by the History Server, so the UI of finished applications can still be inspected:

/opt/spark/sbin/start-history-server.sh --properties-file /opt/spark/conf/spark-defaults.conf

Once running, access http://localhost:18080 to see the list of applications along with their full UI. This is very useful for evaluating job performance after completion or investigating overnight failures.

#Long-Term Dashboards

Event log data can also be reprocessed for historical analysis: the average duration per job, which stages are most often slow, and how resources were used this week. The combination of Prometheus for real-time and the History Server for historical data gives a complete picture.

#Alerting for Failed Jobs and Skew

#Rule-Based Alerts

The goal of observability is early detection. Create alert rules in Prometheus and send notifications through Alertmanager to Slack, email, or PagerDuty:

groups:
  - name: spark
    rules:
      - alert: SparkJobFailed
        expr: spark_app_status == 0
        for: 5m
        labels:
          severity: critical
      - alert: StageDurationHigh
        expr: spark_stage_duration > 1800
        labels:
          severity: warning

spark_app_status == 0 triggers an alert when an application stops with a failed status. Rules like this ensure the team knows before users report it.

#Alerts for Skew and Data Quality

Skew rarely triggers failures, but it always steals time. Create alerts that watch for imbalance:

• A ratio of maximum to median task duration above a certain threshold.
• shuffle read per task that is wildly uneven.
• Data quality checks producing a count(*) below the expected bound.

#Conclusion

Episode 13 equips you with observability skills: the Spark web UI and logs for quick diagnosis, Prometheus and Grafana for real-time metrics, the History Server with event logs for historical inspection, and alerting for early detection of failures and skew.

Key takeaways:

• The Spark UI on port 4040 is the starting point for all diagnosis.
• Event logs must be enabled for the History Server and historical analysis.
• Prometheus stores metrics; Grafana renders them as dashboards.
• Alerts should be few but impactful: failed jobs, slow stages, skew.
• Observability is an investment: it's cheaper to detect early than to investigate after production is disrupted.

In the next episode, episode 14, we'll discuss resource management and deployment — deployment on standalone, YARN, Mesos, and Kubernetes, the Spark on Kubernetes pattern, managing executors, cores, and memory, plus dynamic allocation and workload isolation.