Belajar QA Automation Engineer - Test Maintenance & Flakiness

#Pendahuluan

Setelah di episode 11 kita mengintegrasikan test ke CI/CD, pada episode ini kita membahas musuh terbesar QA automation: flaky tests — tests yang kadang pass kadang fail tanpa perubahan kode.

#Apa itu Flaky Test?

text

Flaky Test: test yang hasilnya tidak deterministik
├── Pass di satu run, fail di run berikutnya
├── Tidak ada perubahan kode yang menyebabkan failure
├── Seringkali timing-dependent
└── Mengurangi confidence pada test suite

#Common Causes

text

Flakiness Root Causes:
├── Timing issues: sleep/tidak ada wait
├── Network: API calls yang lambat
├── State leakage: test sebelumnya mempengaruhi test berikutnya
├── Browser state: cookies, localStorage tidak di-clear
├── Random data: collision antar test
├── Parallel execution: shared resources
└── External dependencies: third-party services

#Debugging Flaky Tests

#Reproduce

bash

# Run test berulang untuk reproduce
npx playwright test --repeat-each=10 login.spec.ts
 
# Jalankan di CI environment
CI=true npx playwright test

#Trace Viewer

bash

# Record trace
npx playwright test --trace on
 
# View trace setelah test selesai
npx playwright show-trace trace.zip

#Debug Mode

typescript

// Tambah debug points
test('flaky test', async ({ page }) => {
  await page.goto('/dashboard');
  console.log('Page loaded:', await page.title());
  await page.pause();  // Pause untuk inspection
  // ...
});

#Fixing Flakiness

#Timing Fixes

javascript

// BAD: manual sleep
await page.waitForTimeout(5000);
 
// GOOD: wait for specific condition
await page.locator('.data-loaded').waitFor({ state: 'visible' });
await page.waitForResponse('**/api/data');

#State Isolation

typescript

// CLEAN: isolated state per test
test.beforeEach(async ({ page }) => {
  // Clear state
  await page.context().clearCookies();
  await page.evaluate(() => localStorage.clear());
 
  // Seed fresh data
  await seedTestData();
});
 
test.afterEach(async ({ page }) => {
  // Cleanup
  await cleanupTestData();
});

#Retry Strategy

typescript

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    screenshot: 'only-on-failure',
    trace: 'on-first-retry',
  },
});

#Maintenance Strategy

text

Maintenance Plan:
├── Monitor flaky rate per test
├── Quarantine consistently flaky tests
├── Fix root causes, bukan tambah retry
├── Regular test suite review
├── Delete obsolete tests
└── Refactor complex tests

Warning

Retry adalah band-aid, bukan solusi. Jika test flaky, cari dan fix root cause-nya. Terlalu banyak retry menyembunyikan masalah nyata.

#Praktik: Fix Flaky Test

typescript

// Before (flaky)
test('load data', async ({ page }) => {
  await page.goto('/dashboard');
  await page.waitForTimeout(3000);  // BAD
  await expect(page.locator('.data')).toBeVisible();
});
 
// After (stable)
test('load data', async ({ page }) => {
  await page.goto('/dashboard');
  await page.locator('.data').waitFor({ state: 'visible' });  // GOOD
  await expect(page.locator('.data')).toBeVisible();
});

#Penutup

Flakiness: timing, network, state, random data, parallel conflicts.
Debug: reproduce dengan repeat, trace viewer, debug mode.
Fix: wait for conditions, state isolation, clean setup/teardown.
Maintenance: monitor flaky rate, quarantine, fix root causes.

Di episode 13 selanjutnya kita akan membahas visual testing — screenshot diff, visual regression, dan pixel snapshots. Sampai jumpa di episode 13!