Learn Selenium - Operational Readiness & Runbooks

#Introduction

A test suite running in CI is a production system — and production systems need operational procedures. Episode 19 covers operational readiness: runbooks for handling broken tests, strategies for facing maintenance windows and environment drift, a disciplined failure triage process, and test ownership within the team.

Many teams build their suite with enthusiasm, then lose that enthusiasm once the suite starts demanding routine attention. The difference is in procedure: teams with runbooks and ownership rules handle failures calmly, while teams without procedures panic at every red CI.

#Runbooks for Broken Tests

#Runbook Structure

A runbook is a step-by-step document for responding to incidents. For broken tests, create a template that answers three questions: what the symptoms are, how to diagnose, and how to recover.

Judul: Test X gagal di CI
1. Lihat artifact screenshot dari job.
2. Cek log test untuk langkah terakhir yang berhasil.
3. Cek versi browser yang dipakai vs versi driver.
4. Reproduksi lokal dengan perintah yang sama.
5. Jika berulang, buat ticket dan assign ke pemilik test.

The template above is an example runbook structure. The steps run from the cheapest evidence — screenshots and logs — to full investigation, so anyone can follow them without deep experience.

#Runbooks in the Repository

Store runbooks in the repository, for example under docs/runbooks/, so they're updated alongside the code. A runbook living in a private document drawer will never be read when needed.

#Maintenance Windows and Environment Drift

#Maintenance Windows for Tests

Tests also need scheduled maintenance time. Create a recurring agenda — for example weekly — for cleaning up obsolete tests, updating changed selectors, and verifying visual baselines. Without a maintenance window, dirt accumulates until it becomes a big project.

#Environment Drift

Environment drift happens when the test environment differs from production or changes silently: browser versions change, test data expires, or staging is reset. Detect drift by keeping an environment record:

google-chrome --version
chromedriver --version

google-chrome --version and chromedriver --version record the versions in use. Note both numbers in the log on every run — version changes are one of the most common causes of broken tests unrelated to code.

#Managing the Test Suite and Triaging Failures

#Triage Priority

Not all failures are equal. Apply layered triage:

1. Infrastructure: grid down, browser fails to start — fix the environment first.
2. Automation: selectors broken, waits insufficient — correct the test.
3. Product: application behavior changed — confirm with the product team.

Concise triage script

def klasifikasi_gagal(pesan):
    if "session not created" in pesan:
        return "infrastruktur"
    if "no such element" in pesan:
        return "automasi"
    return "produk"

The klasifikasi_gagal(pesan) function is an example of simple triage automation based on error messages. Classifying failures before hunting details keeps the team's effort focused.

#Triage Metrics

Track the number of failures per category every week. If the "automation" category dominates, the suite needs quality improvements; if "product" dominates, the application changes too fast — both need different approaches.

#Recovery for Browser Compatibility

#Standard Recovery Steps

Browsers auto-updating without permission is the classic cause of broken tests. The recovery runbook for this is short and clear:

google-chrome --version
pip show selenium | grep -i version

If a test breaks after a browser update, check compatibility: the driver must match the browser (episode 3), and the Selenium API used must be supported by that browser version. Selenium Manager usually resolves this automatically; for special cases, pin the browser version in CI.

#Pinning Browser Versions in CI

For maximum stability, use CI images with pinned browser versions, like the official Selenium images already paired with drivers:

services:
  chrome:
    image: selenium/standalone-chrome:132.0

selenium/standalone-chrome:132.0 pins both the Chrome and driver versions together. Pinning buys time — the team gets room to adapt before the next update.

#Team Ownership and Test Maintenance

#One Test, One Owner

Every test should have a clear owner. This isn't bureaucracy — it prevents the "tragedy of the commons" where everyone assumes a broken test is someone else's problem. An ownership map is simple enough:

• E-commerce suite: checkout team.
• Login and account suite: identity team.
• Test infrastructure: platform/QA team.

#Making Maintenance a Team Rhythm

Schedule maintenance as a team activity, not a punishment for one person. Review the most frequently failing tests, study their patterns, and fix them together. A suite maintained together outlives a suite cared for by a single person who leaves.

#Conclusion

Episode 19 turns your suite from a "collection of scripts" into a "system with procedures": runbooks for broken tests, maintenance window and anti-drift strategies, layered failure triage, browser compatibility recovery, and clear test ownership within the team.

Key takeaways:

• Runbooks provide ordered steps for responding to broken tests.
• Record browser and driver versions to detect environment drift.
• Triage failures: infrastructure first, then automation, then product.
• Pin browser versions in CI for stability, and match them when broken.
• Every test has an owner; maintenance is a scheduled team activity.

In episode 20 next, we'll cover real-world use cases and patterns — e-commerce checkout use cases, login flows, and dashboard interactions, test design patterns for end-to-end automation, the difference between component-level and full UI tests, and how to prioritize tests for business-critical flows.