Learn Apache Spark - Machine Learning & MLlib

#Introduction

Apache Spark isn't just an ETL and SQL engine — it also contains MLlib, a scalable machine learning library. With MLlib, you can train classification and regression models on terabyte-scale datasets without moving the data to another system.

Why does MLlib matter? Most production ML pipelines are front-ended by Spark: data is cleaned and prepared in Spark, models are trained with MLlib, and the results are used for batch or real-time predictions. Understanding MLlib means understanding how machine learning runs at enterprise scale.

This episode covers four things: the pipeline API concept with transformers and estimators, feature engineering, model training and tuning, and use cases for classification, regression, clustering, and recommendation.

#Introduction to MLlib and the Pipeline API

#The Pipeline Concept

The Pipeline in MLlib follows a pattern similar to sklearn: a sequence of stages that run in order. There are two kinds of stages:

• Transformer: transforms a DataFrame, usually by adding columns — e.g. VectorAssembler, StringIndexer.
• Estimator: learned from data (fit) and produces a model that then becomes a transformer — e.g. LogisticRegression, RandomForestClassifier.

data → StringIndexer → VectorAssembler → LogisticRegression → prediction
       (transform)       (transform)       (estimator → model)

The advantage of pipelines: the entire fit-and-transform flow is wrapped as a single object, so it can be saved, loaded, and rerun consistently.

#Building a Pipeline

Simple pipeline

from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler, StringIndexer
 
indexer = StringIndexer(inputCol="kategori", outputCol="kategori_idx")
assembler = VectorAssembler(
    inputCols=["umur", "jumlah", "kategori_idx"],
    outputCol="features")
lr = LogisticRegression(featuresCol="features", labelCol="churn")
 
pipeline = Pipeline(stages=[indexer, assembler, lr])

Pipeline(stages=[...]) defines the flow. No computation happens at this stage — the model is only learned when pipeline.fit() is called.

#Feature Engineering: Transformers and Estimators

#Basic Transformers

Feature engineering is the process of turning raw columns into features the model can use. The most common transformers:

• StringIndexer: converts string categories into numeric indices.
• OneHotEncoder: converts category indices into binary representations.
• VectorAssembler: combines multiple columns into a single feature vector.
• StandardScaler: normalizes the scale of numeric features.

Complete feature engineering

from pyspark.ml.feature import OneHotEncoder, StandardScaler
 
encoder = OneHotEncoder(inputCol="kategori_idx", outputCol="kategori_vec")
scaler = StandardScaler(inputCol="features_raw", outputCol="features")
 
pipeline = Pipeline(stages=[indexer, encoder, assembler_raw, scaler, lr])

StandardScaler matters when features have very different ranges — for example age in the tens and transaction counts in the millions. Without scaling, models like regression get dominated by the largest-scale features.

#Estimators and Models

After fitting, an estimator produces a model that can be used for transformations. Classic models available in MLlib include:

• LogisticRegression: binary and multinomial classification.
• RandomForestClassifier and GBTClassifier: tree ensembles.
• LinearRegression: regression.
• KMeans: clustering.
• ALS: collaborative filtering for recommendation.

#Model Training, Tuning, and Evaluation

#Training and Evaluating a Model

Train the pipeline, then evaluate with metrics:

Fit, predict, and evaluate

model = pipeline.fit(data_latih)
prediksi = model.transform(data_uji)
 
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
evaluator = MulticlassClassificationEvaluator(labelCol="churn", metricName="accuracy")
print("Akurasi:", evaluator.evaluate(prediksi))

model.transform(data_uji) runs the entire pipeline — indexing, encoding, scaling, all the way to prediction — in a single call. The evaluator measures model performance on data it has never seen.

#Tuning with CrossValidator

Finding the best hyperparameters should be done systematically with CrossValidator or TrainValidationSplit:

Tuning with CrossValidator

from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
 
grid = ParamGridBuilder() \
    .addGrid(lr.regParam, [0.01, 0.1]) \
    .addGrid(lr.maxIter, [10, 50]) \
    .build()
 
cv = CrossValidator(
    estimator=pipeline,
    estimatorParamMaps=grid,
    evaluator=evaluator,
    numFolds=3)
 
cv_model = cv.fit(data_latih)

cv.fit(data_latih) trains every hyperparameter combination on several folds and selects the best one based on the evaluator. Keep in mind: tuning trains the model repeatedly, so make sure the dataset isn't too large, or use a sample first.

#Use Cases: Classification, Regression, Clustering, and Recommendation

#Classification and Regression

Classification predicts categorical labels (churn, spam, customer type); regression predicts continuous values (price, sales, time). Both follow the same pipeline pattern — only the model and evaluator differ.

#Clustering with KMeans

To find hidden groups without labels, use clustering:

KMeans clustering

from pyspark.ml.clustering import KMeans
 
kmeans = KMeans(featuresCol="features", k=4, seed=42)
model_k = kmeans.fit(data_fitur)
hasil_k = model_k.transform(data_fitur)

KMeans(featuresCol="features", k=4) splits the data into four clusters. The cluster count k needs exploration — one way is to use the elbow technique by looking at the cost (sum of squared distances) for various values of k.

#Recommendation with ALS

For recommendation systems, MLlib provides ALS (Alternating Least Squares) — collaborative filtering based on user-item interactions:

ALS for recommendations

from pyspark.ml.recommendation import ALS
 
als = ALS(
    userCol="user_id", itemCol="produk_id", ratingCol="rating",
    coldStartStrategy="drop")
 
model_als = als.fit(interaksi)
rekomendasi = model_als.recommendForAllUsers(5)

recommendForAllUsers(5) produces the top five recommendations for every user. coldStartStrategy="drop" is important so new users or items without ratings don't produce null predictions.

classification   → LogisticRegression, RandomForest
regression       → LinearRegression, GBTRegressor
clustering       → KMeans, BisectingKMeans
recommendation   → ALS (collaborative filtering)

#Conclusion

Episode 11 equips you with distributed machine learning: the pipeline API cleanly separates transformers and estimators, feature engineering turns raw data into feature vectors, CrossValidator performs systematic tuning, and the various models handle classification, regression, clustering, and recommendations.

Key takeaways:

• Pipelines combine transformers and estimators into one flow.
• VectorAssembler and StandardScaler are the foundation of feature engineering.
• Always evaluate on data that didn't train the model.
• CrossValidator selects hyperparameters systematically.
• ALS handles recommendation without manual feature engineering.

In the next episode, episode 12, we'll discuss security and authentication — securing the cluster with TLS/SSL, authentication and authorization for Spark jobs, securely managing data source credentials, and integration with Kerberos and LDAP.