Learn Cypress - Operational Readiness & Runbooks

#Introduction

A healthy suite is not created in one shot — it is maintained like a production system. Episode 19 covers the operational side: runbooks for flaky tests and environment drift, incident response for failing CI runs, monitoring coverage and health, and ownership assignment and maintenance routines.

By the end of this episode, your team has procedures — not just reactions — when the suite starts misbehaving.

#Runbooks for Flaky Tests and Environment Drift

#Defining a Runbook

A runbook is a step-by-step document for expected situations. For flaky tests, start with diagnosis:

1. Reproduksi: jalankan spec 3x, catat pola kegagalan.
2. Periksa timing: adakah cy.wait(ms) atau elemen perantara?
3. Periksa data: adakah ketergantungan pada data yang berubah?
4. Perbaiki akar masalah, bukan hanya menambah retries.

The line 1. Reproduksi: jalankan spec 3x, catat pola kegagalan. is the mandatory first step. Runbooks turn flaky fixing from guessing into a process anyone can follow.

#Handling Environment Drift

Environment drift happens when the test environment differs from production or changes without being noticed. Verify the environment at the start of the run:

Checking the environment in CI

describe("Sanity check", () => {
  it("environment siap untuk test", () => {
    cy.request("GET", "/api/health").then((resp) => {
      expect(resp.status).to.eq(200);
    });
  });
});

cy.request("GET", "/api/health") confirms the backend is alive before the main suite. Sanity tests like this catch drift early and make suite failures easier to read.

#Incident Response for Failing CI Runs

#Quick Triage

When CI goes red, the first thing is not fixing — it is classifying. Make a clear list of categories:

• Flaky: fails occasionally without code changes.
• Regression: fails consistently after the latest change.
• Environment: fails because of the backend, seed, or configuration.

Distinguishing the category decides the next step: flaky is flagged for investigation, regression is muted or handled immediately, environment is restored to a normal state. Failures documented with a category are much easier to handle.

#Controlled Muting

When a run blocks the team and work must continue, mute it with a recorded reason:

Muting a test with a reason

it.skip("regresi pada daftar produk - menunggu perbaikan #1234", () => {
  cy.visit("/katalog");
});

it.skip(...) disables the test while keeping the issue number. Mute only for a short time; add it to the fix backlog and monitor so it is not forgotten.

#Maintaining Test Coverage and Health

#Monitoring Metrics

Suite health is measured with consistent metrics:

Pass rate: 98% (target: min 95%)
Durasi: 6 menit (target: max 10 menit)
Flaky minggu ini: 3 (target: mendekati 0)

Pass rate: 98% (target: min 95%) provides an explicit target. Without targets, metrics cannot be debated. Also set targets for duration and the number of flaky tests so health is measurable.

#Keeping Coverage Meaningful

Measured coverage is the number of tests per important flow, not just a percentage of code lines. Review periodically: which flows still have no tests, which tests are duplicates, and which flows are so rarely used they are not worth maintaining. Meaningful coverage beats impressive numbers.

#Team Ownership and Maintenance Routines

#Ownership per Module

Tests are healthiest when they have an owner. Split the suite into modules and assign an owner:

auth-flow:
  owner: tim-frontend
  specs:
    - cypress/e2e/auth/**
checkout:
  owner: tim-commerce
  specs:
    - cypress/e2e/checkout/**

auth-flow: defines a module along with its owner. When an auth test fails, the frontend team knows who to contact. Ownership prevents the "nobody's responsibility" status.

#Maintenance Routines

Schedule routine maintenance: weekly flaky test review, monthly selector and helper audit, and a quarterly review of the whole suite against application changes. Consistent routines keep the suite healthy before problems pile up.

#Closing

Episode 19 made testing a system that is operated: runbooks for flakiness and drift, triage for failing CI, health metrics with targets, and clear ownership and maintenance routines.

Key takeaways:

• Runbooks turn flaky fixing into a process anyone can follow.
• Classify failures: flaky, regression, or environment.
• A sanity check at the start of a run catches drift early.
• Set targets for pass rate, duration, and the number of flaky tests.
• Assign ownership per module and schedule routine maintenance.

In the next episode, episode 20, we will cover real-world use cases and patterns — case studies of ecommerce, SaaS onboarding, and dashboard flows, test design and prioritization best practices, balancing unit, integration, and E2E tests, and end-to-end reliability patterns.