Learn Selenium - Test Stability & Flakiness

#Introduction

Nothing drains a team's trust in automation faster than flaky tests — tests that sometimes pass and sometimes fail with no changes at all. A team constantly haunted by random failures starts ignoring CI results, and the suite loses its entire value. Episode 13 covers eradicating flakiness at its root.

We'll dissect the sources of flakiness, build reliable selectors, apply proper retry and flaky detection, and monitor suite stability as a metric guarded like any other.

#Sources of Flakiness

Before fixing, know the causes. The five most common sources of flakiness in Selenium automation:

• Timing: the element isn't there yet when looked up — solved with proper waits (episode 5).
• Weak selectors: locators that match many elements or change with layout.
• Shared state: tests affecting each other through shared data or sessions.
• Environment: unstable browsers, drivers, or machine resources.
• Execution order: tests that only pass if they run after other tests.

timing -> selector -> shared state -> environment -> test order

timing -> selector -> shared state -> environment -> test order is the recommended checking order: inspect the most frequent causes first.

#Reliable Selectors

#Locator Selection Rules

Selector quality determines most of a suite's stability. These are the rules we apply in every project:

• Prioritize id and data-testid created specifically for testing.
• Avoid text-based locators that easily change with new copy.
• Avoid indexes like div[2] that are brittle to structure changes.
• Don't over-specify — overly specific locators break just as easily.

Strong vs weak selectors

# Strong: test-specific attribute
driver.find_element("css selector", "[data-testid='tombol-beli']")
 
# Weak: position and generic classes
driver.find_element("css selector", "div.wrapper > div:nth-child(2) > button.btn")

The [data-testid='tombol-beli'] pattern is far more stable than div.wrapper > div:nth-child(2) > button.btn. If the product team is willing to add data-testid, that investment pays off many times over in suite stability.

#Consistency Across the Suite

The same selector for the same element should be used in all tests. This isn't just tidiness — it prevents two tests from referencing the same element in two different ways, so one layout change only breaks a single point.

#Retry Strategies and Flaky Detection

#Retry as a Temporary Safety Net

Retry isn't a permanent solution, but it can save the pipeline while the root cause is hunted. The pytest-rerunfailures plugin provides per-test retries:

pip install pytest-rerunfailures

After pip install pytest-rerunfailures, mark tests that need retries:

Retry per test

@pytest.mark.flaky(reruns=2, reruns_delay=1)
def test_checkout():
    ...

@pytest.mark.flaky(reruns=2, reruns_delay=1) retries the test up to twice with a one-second pause. Important: every test needing a retry must have a fix ticket — retry without investigation only hides the problem.

#Detecting Flaky Tests

The most effective way to detect flakiness: run the test repeatedly and watch for occasional failures:

pytest tests/test_checkout.py -n 4 --count 3

pytest tests/test_checkout.py -n 4 --count 3 (with the pytest-repeat plugin) runs the test many times. Tests that occasionally fail are flaky candidates that must be investigated before they pollute the suite.

#Monitoring Execution Stability

Stability is a measurable metric. Track three numbers on every CI run:

• Pass rate: the percentage of passing tests.
• Flaky rate: tests whose results change between runs.
• Duration: the time the full suite takes.

Store these numbers in reports or a simple dashboard. The threshold we recommend: pass rate above 99% and flaky rate near zero — when it slips, the team's priority is stabilization, not adding new tests.

#Best Practices for a Stable Automation Suite

A summary of habits that keep a suite stable long-term:

• Test isolation: each test prepares its own data, never depending on other tests.
• Mandatory teardown: close the driver and clean up data, whatever the test result.
• Use explicit waits for important interactions, not static sleeps.
• Combine with API setup (episode 16) to skip slow UI.
• Schedule flaky checks as a weekly team ritual.

Simple isolation fixture

@pytest.fixture
def data_test():
    pengguna = buat_pengguna_unik()
    yield pengguna
    hapus_pengguna(pengguna.id)

buat_pengguna_unik() guarantees every test has its own data. This create-use-delete pattern prevents the data collisions that breed flakiness.

#Conclusion

Episode 13 changes how teams view flaky tests: not random noise to tolerate, but bugs to hunt. You can now recognize the five sources of flakiness, build reliable data-testid-based selectors, apply retry as a temporary net, and monitor pass rate as a team metric.

Key takeaways:

• Five sources of flakiness: timing, selector, shared state, environment, test order.
• id and data-testid selectors are far more stable than position or text.
• Retry is temporary handling, not a replacement for fixing the root cause.
• Test data isolation prevents collisions between tests.
• Monitor pass rate and flaky rate as team stability metrics.

In episode 14 next, we'll cover cross-browser testing strategies — running tests in Chrome, Firefox, Edge, and Safari, compatibility testing patterns, managing browser-specific quirks, and using cloud browser farms like BrowserStack and Sauce Labs.