Learn k6 - Resilience, Retry, and Failure Analysis

#Introduction

In episode 14, you connected k6 to the observability ecosystem: metrics flow to Grafana via InfluxDB and Prometheus, so system performance can be seen as time-series data. You can now see problems. This episode 15 teaches how to respond to them correctly — two skills equally important in real operations.

First, we understand the fundamental difference between a flake (transient disturbance) and a fail (real failure). Second, we design retry with backoff — behavior every well-behaved client must have when the server is under pressure. Finally, we break down the k6 HTTP time metrics layer by layer to find exactly where time is lost. Let's get started.

#Flake vs Fail: Distinguishing Transient Disturbances from Real Failures

Imagine calling a call center. Sometimes you hear a busy tone because all operators are handling other calls — that's a flake: a disturbance that resolves on its own, and calling back a few seconds later will most likely succeed. If the busy tone is heard every time, or the phone is picked up but immediately hung up, that's a fail: a structural problem that won't disappear just because you call back.

In load testing, this distinction determines the team's reaction:

• Flake — non-deterministic, appears occasionally, and disappears when repeated. Examples: one in a thousand requests returns 500 due to an autoscaler cold start, 429 due to a momentary rate limiter, or a timeout caused by a network hiccup.
• Fail — consistent and reproducible. Examples: all requests to a new endpoint fail with 500 ever since the latest code was deployed, or a database query always exceeds its timeout when the payload crosses a certain size.

Why is this distinction important? If you treat a flake as a fail, the team will hunt ghosts — replacing load balancers, adding nodes, changing configuration — when the root cause was just one blinking connection. Conversely, if a fail is treated as a flake, a real bug slips into production. The rule: retry flakes, investigate fails, and make the error rate (not a single request) your decision signal.

#Retry and Backoff Strategy

Retry is about redeeming a flake: try again, but with discipline. There are three decisions to make: which statuses may be retried, how long to wait between attempts, and how many attempts at most.

#Statuses Worth Retrying

Never retry all failures. The correct grouping:

• Retry: 429 Too Many Requests, all 5xx, and network/timeout errors. This category signals that the server is busy or recovering — retrying is reasonable.
• Don't retry: 4xx other than 408 and 429 (for example 400, 401, 403, 404). These are your own request errors; repeating with the same payload only adds server load and slows the test.

#Simple Backoff and Exponential Backoff

Backoff is the pause between attempts. Simple backoff waits a fixed duration, for example one second each time. Exponential backoff doubles the pause on each attempt: 200 ms, then 400 ms, then 800 ms. The analogy is starting a car engine on a cold morning: pressing the starter continuously only drains the battery, while waiting a moment and trying again is the most effective. Exponential backoff also prevents the thundering herd — tens of thousands of requests that just failed all retrying in the same second, making a recovering server fall over again.

#Retry Helper with Backoff

import { sleep } from 'k6';
 
const DEFAULT_MAX_ATTEMPTS = 4;
const DEFAULT_BASE_DELAY_MS = 200;
 
export function retryWithBackoff(fn, options = {}) {
  const maxAttempts = options.maxAttempts || DEFAULT_MAX_ATTEMPTS;
  const baseDelayMs = options.baseDelayMs || DEFAULT_BASE_DELAY_MS;
 
  let lastResponse;
 
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    lastResponse = fn();
 
    if (!isRetryable(lastResponse.status)) {
      return lastResponse;
    }
 
    if (attempt === maxAttempts) {
      return lastResponse;
    }
 
    const delayMs = baseDelayMs * Math.pow(2, attempt - 1);
    sleep(delayMs / 1000);
  }
 
  return lastResponse;
}
 
function isRetryable(status) {
  return status === 429 || status >= 500;
}

Notice the flow: the callback function is executed as one attempt, its result is checked against the retryable statuses, and the sleep pause actually runs between attempts — not just as a number computation. When all attempts fail, the helper returns the last response instead of throwing an error, so check() can still evaluate the result and thresholds keep working. Usage inside a scenario:

import { check } from 'k6';
import http from 'k6/http';
import { retryWithBackoff } from './lib/retry.js';
 
export default function () {
  const response = retryWithBackoff(() => {
    return http.get('https://api.example.com/checkout');
  });
 
  check(response, {
    'checkout 200': (r) => r.status === 200,
  });
}

#Failure Analysis: Breaking Down Root Causes from Metrics

When a threshold is violated, many people's first reflex is to guess: is it the database? the load balancer? the new code? Guessing is expensive. k6 gives you a better tool: the timing breakdown metric that splits an HTTP request's duration into sequential phases. By comparing which phase swells, you turn guesses into a diagnosis. When k6 run script.js ends with a violated threshold, this is the map you should open first:

The first three phases (blocked, connecting, tls_handshaking) happen before the server has processed anything — there, time is spent on the road. The last three phases (sending, waiting, receiving) are time related to the server and payload. In practice, http_req_waiting is almost always the culprit: a slowing application makes wait time swell, and as a knock-on effect http_req_blocked also rises because connections pile up queuing in the pool — a classic phenomenon often misdiagnosed as a network problem when the root is in the application.

#Reading Patterns from the Breakdown

You can stream the http_req_* metrics to Grafana (episode 14) and compare each phase's p95 in a single panel. Patterns to recognize:

• waiting and blocked rising together → the application is slowing down and connections are queueing. Aim debugging at the application server and database.
• connecting or tls_handshaking high on their own → a problem in the network path or handshake; check DNS, region, and certificates.
• receiving high with waiting normal → the response body is swelling; check whether the client requests too much data.
• blocked high without connecting rising → your test machine's resources are exhausted; reduce VUs or scale up the machine.

Combined with the error rate broken down by status code — for example 429 dominating means rate limiting, 502 dominating means a problematic gateway or proxy — you have enough of a map to find the layer to investigate, without guessing.

#Common Mistakes in Resilience and Failure Analysis

#Conclusion

Episode 15 gave you two weapons for facing degradation: discipline — distinguishing flakes from fails, designing retries with backoff and attempt limits, choosing which statuses are worth retrying — and clarity — reading the breakdown of the http_req_blocked, connecting, tls_handshaking, sending, waiting, and receiving metrics to find the layer that is actually problematic.

The good news: all of these strategies can run without human supervision. In episode 16, we put k6 into CI/CD and automation workflows: GitHub Actions, GitLab CI, and Jenkins, with exit code 99 as the universal language that fails the build when a threshold is violated. Your performance will soon become an automated decision, not an opinion. See you there!