Learn Playwright - Operational Readiness & Runbooks

#Introduction

A healthy suite isn't just about green tests today — it's about how the team reacts when tests fail, who's responsible, and how to recover after a problem. This episode 19 covers operational readiness: runbooks, ownership, and maintenance processes that make a suite manageable in the long term.

Like production operations, a test suite needs procedures. A runbook answers "what do I do right now?" when tests are flaky, when there's environment drift, or when a browser fails. Ownership answers "who resolves this?" And maintenance answers "when do we care for the suite before it breaks?" We'll put all three together one by one.

#Runbooks for Flaky Tests, Environment Drift, and Browser Failures

#Flaky Test Runbook

A flaky test is one that sometimes passes and sometimes fails with no code change. Here's the step-by-step runbook:

1. Grab the trace and artifacts from CI — don't guess.
2. Check whether the failure relates to timing or a locator.
3. Open the trace viewer and look at the failing step.
4. If it's a locator issue, replace it with getByRole or getByTestId.
5. If it's a timing issue, replace waitForTimeout with waiting for a condition.
6. Add a temporary retry, but still flag it for root cause investigation.

[ ] Grab the trace from CI
[ ] Identify the failing step
[ ] Check the locator and timing
[ ] Fix the root cause, not the symptoms

#Environment Drift Runbook

Environment drift happens when CI differs from local — Node, browser, or dependency versions aren't in sync. Detection and steps:

• Compare npx playwright --version in CI and locally.
• Make sure the lockfile (package-lock.json or bun.lock) is committed.
• Use the official Playwright action or image, which already contains the dependencies.
• Make sure seed data and environment variables are identical.

#Browser Failure Runbook

A browser that can't launch is usually due to missing system dependencies on Linux CI. The fix:

npx playwright install --with-deps

npx playwright install --with-deps installs the browsers along with system dependencies. If the CI runner already uses the official Playwright image, these dependencies are already there — so choose an image matching your Playwright version.

#Managing Test Suite Health and Team Ownership

#Suite Health Metrics

Suite health can be measured with simple metrics monitored regularly:

• Flakiness rate: the percentage of tests that fail and then pass after a retry.
• Total duration: the time the whole suite takes to run.
• Skip/fixme count: how many tests are inactive and why.
• Feature coverage: what percentage of critical business flows have tests.

npx playwright test --reporter=json --output=report.json

--reporter=json produces structured data that a script can process to monitor health metrics automatically.

#Suite Ownership

Every test suite needs a clear owner — not "everyone," which really means no one. Assign ownership per area: the frontend team owns UI tests, the backend team owns API tests, and one coordinator maintains the suite's configuration and infrastructure.

#Code Review for Tests

Treat tests like production code: every test change must go through review. This prevents changes that lower quality — for example, removing an assertion to make a test pass, or adding a waitForTimeout to cover up a problem.

#Recovery Strategy When Test Baselines Change

#Baselines Changed Intentionally

When the design changes, visual baselines (episode 16) must be updated. A safe procedure:

1. Run the visual tests and inspect every diff.
2. Confirm the changes actually match the new design.
3. Update the baselines with --update-snapshots.
4. Review the diff in the PR so everyone sees what changed.

#Baselines Changed Unintentionally

If a diff appears with no design change, it's an alarm — don't update the baseline to cover it up. Investigate the cause: maybe fonts changed, dynamic content got rendered, or the rendering environment differs.

#Handling Major Changes

For major changes (a full page redesign), freeze the visual tests temporarily, complete the change, then regenerate the baselines from the correct state. Document the freeze period so the team understands why visual tests didn't run during that time.

#Maintenance Schedule and Cleanup Tasks

#A Regular Maintenance Schedule

A suite needs periodic care, not just when it breaks. A healthy schedule:

• Weekly: check the flakiness rate and frequently failing tests.
• Per sprint: audit skips and fixmes, add tests for new features.
• Per release: review suite duration, update visual baselines, remove obsolete tests.

#Cleaning Up Obsolete Tests

Removed features leave behind dead tests. Delete obsolete tests together with their features — tests pointing at features that no longer exist only add runtime and confusion.

[ ] Remove tests for removed features
[ ] Update locators that reference old elements
[ ] Verify fixtures and helpers still in use
[ ] Update snapshot baselines after design changes

#Scheduling Artifact Cleanup

Test artifacts (screenshots, videos, traces) quickly fill up storage. Set retention in CI — GitHub Actions can set retention-days, and the local test-results folder can be cleaned automatically:

rm -rf test-results playwright-report

The command rm -rf test-results playwright-report cleans up local test results. Get in the habit of running it periodically so the workspace doesn't bloat with old artifacts.

#Closing

Episode 19 equipped you with the operational side of testing: runbooks for the three most common failures, metrics and ownership for keeping the suite healthy, recovery procedures when baselines change, and a maintenance and cleanup schedule that prevents silent decay.

Key takeaways:

• Runbooks speed up responses to flakiness, environment drift, and browser failures.
• npx playwright install --with-deps resolves most Linux CI browser failures.
• Monitor suite health metrics and assign ownership per area.
• Update baselines only when the change is intentional and verified.
• Schedule periodic maintenance and clean up artifacts and obsolete tests.

In the next episode we'll discuss real-world use cases and patterns — real examples of e-commerce checkout, signup flows, and dashboard workflows, test design patterns for end-to-end automation, balancing UI tests with API and component tests, and prioritizing tests for critical business paths.