Learn Apache Spark - Cross-System Integration

#Introduction

Spark never works alone. In a real data architecture, Spark sits at the center of an ecosystem: reading from Kafka, writing to Cassandra, sending documents to Elasticsearch, and populating data warehouses. Episode 18 covers how Spark connects to these systems.

This integration ability is what makes Spark the "glue of data engineering." An engineer who masters integration can design a complete data flow — from source, through Spark, to consumer — without depending on special tools for every pair of systems.

This episode covers four topics: connectors to Kafka, Cassandra, and Elasticsearch; Spark as an ETL engine for data lakes and warehouses; BI tool integration; and data ingestion and CDC patterns.

#Connecting Spark to Kafka, Cassandra, and Elasticsearch

#Kafka: The Streaming Backbone

Kafka is the most common message queue system in the Spark ecosystem. As a source, Spark reads via Structured Streaming as in episode 10. As a sink, streaming results are sent back to a topic:

Reading and writing Kafka

from pyspark.sql import functions as F
 
kafka_in = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "orders") \
    .load()
 
hasil = kafka_in.selectExpr("CAST(value AS STRING) AS value")
 
kafka_out = hasil.writeStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("topic", "orders-enriched") \
    .option("checkpointLocation", "data/ckpt-kafka") \
    .start()

selectExpr("CAST(value AS STRING) AS value") ensures the value column is in string format before being written back. Kafka provides an exactly-once guarantee when combined with correct checkpoints.

#Cassandra: Fast Writes

Cassandra is a distributed wide-column database popular for write-heavy workloads. Spark integrates through the DataStax connector:

Writing to Cassandra

df.write \
    .format("org.apache.spark.sql.cassandra") \
    .option("keyspace", "analitik") \
    .option("table", "ringkasan") \
    .mode("append") \
    .save()

df.write.format("org.apache.spark.sql.cassandra") writes a batch to Cassandra. This connector manages token ranges and batch sizes automatically — keep in mind that performance depends heavily on the primary key design and partition count of the Cassandra table.

#Elasticsearch: Indexing for Search

Elasticsearch is used for text search and dashboards. Spark sends documents through the connector:

Sending to Elasticsearch

df.write \
    .format("org.elasticsearch.spark.sql") \
    .option("es.nodes", "es-node:9200") \
    .option("es.resource", "analitik") \
    .mode("overwrite") \
    .save()

es.resource determines the destination index. For streaming pipelines, the Elasticsearch sink is also supported through the same connector.

#Spark as an ETL Engine for Data Lakes and Warehouses

#ETL to the Data Lake

The most classic role: Spark pulls data from various sources, cleans and transforms it, then writes it to the data lake in Parquet or Delta format:

source (DB, API, Kafka) → Spark (extract + transform) → data lake (Parquet/Delta)

Spark's advantages for ETL: one engine for batch and streaming, connectors to almost every source, and transformations that can be tested repeatedly.

#Loading into a Data Warehouse

For modern warehouses like Snowflake or BigQuery, Spark writes through official connectors:

Writing to Snowflake

df.write \
    .format("net.snowflake.spark.snowflake") \
    .option("sfUrl", "...") \
    .option("sfUser", "...") \
    .option("sfDatabase", "dw") \
    .option("sfSchema", "public") \
    .save()

A lighter alternative: Spark writes Parquet to staging in object storage, then the warehouse runs COPY INTO — a pattern that reduces the load on direct connectors and leverages the warehouse's native speed.

#Integration with BI Tools and Analytics Platforms

#Connecting BI to Spark Results

BI tools don't read Spark data directly. The common flows:

• JDBC/Thrift: run a Spark Thrift Server so BI tools can query Spark data with standard SQL.
• Warehouse connectors: BI reads from the warehouse that Spark populated.
• Open formats: BI reads Parquet/Delta directly from the data lake via engines like DuckDB or Trino.

/opt/spark/sbin/start-thriftserver.sh

start-thriftserver.sh starts a JDBC/ODBC endpoint on port 10000 — tools like Tableau or Superset can then connect to it like a regular database.

#Access Management Patterns

At scale, BI access should go through a warehouse or open engine, not ad-hoc Spark sessions — so resources stay isolated and no analyst accidentally runs a giant query on the production cluster.

#Data Ingestion and CDC Patterns

#CDC with Change Data Capture

CDC (Change Data Capture) captures database changes (insert, update, delete) and streams them for processing. The common pattern:

1. The database emits changes via binlog/WAL to Kafka.
2. Spark reads the change stream with Structured Streaming.
3. Spark applies the changes to the target (data lake or warehouse) with MERGE.

Applying CDC with merge

from pyspark.sql import functions as F
 
perubahan = spark.readStream.format("kafka").option("subscribe", "db-changes").load()
target = spark.read.format("delta").load("data/target_delta")
 
def proses_batch(df, epoch_id):
    df.createOrReplaceTempView("perubahan")
    target.sparkSession.sql("""
        MERGE INTO data/target_delta AS t
        USING perubahan AS s ON t.id = s.id
        WHEN MATCHED AND s.op = 'delete' THEN DELETE
        WHEN MATCHED THEN UPDATE SET *
        WHEN NOT MATCHED THEN INSERT *
    """)
 
query = perubahan.writeStream.foreachBatch(proses_batch).start()

foreachBatch(proses_batch) allows full batch logic (including MERGE) to be applied to each micro-batch — the most flexible pattern for CDC in Spark.

#Idempotency and Ordering

Database changes have an important order. When applying CDC, watch out for:

• Idempotency: applying the same change twice must produce the same result.
• Ordering: make sure events for one key are processed in sequence — group by key and add timestamps.
• Exactly-once: use checkpoints and transactional sinks like Delta.

#Conclusion

Episode 18 equips you with integration skills: the Kafka, Cassandra, and Elasticsearch connectors link Spark to consumer systems, Spark's role as an ETL engine populates data lakes and warehouses, BI tools reach data through a Thrift Server or warehouse, and CDC patterns keep data fresh at the target.

Key takeaways:

• Kafka serves as both streaming source and sink with exactly-once guarantees.
• Cassandra and Elasticsearch are accessed through format-specific connectors.
• Spark populates data lakes and warehouses; warehouses serve BI.
• The Thrift Server exposes Spark SQL to standard BI tools.
• CDC streams database changes and processes them with idempotent merges.

In the next episode, episode 19, we'll discuss operational readiness and runbooks — writing runbooks for job failure and recovery, incident response for clusters and data corruption, backing up configuration and artifacts, and chaos testing to verify pipeline resilience.