Learn Observability with the LGTM Stack - Performance Testing & Benchmarking

#Introduction

Building observability in a small environment is easy. The question is always the same: will this stack hold up at 10 times the traffic? The answer can't be guessed — it must be tested.

This episode covers application load testing with k6, benchmarking the observability stack with avalanche and loggen, establishing performance baselines, and capacity testing with stress, soak, and spike scenarios.

#Load Testing with k6

#k6 and Grafana

k6 is an open-source load testing tool integrated directly with the Grafana ecosystem:

• Prometheus remote write from k6: test metrics are sent to Mimir.
• Real-time performance dashboards: test results appear live in Grafana.
• Correlating load with system metrics: compare synthetic load with system metrics.

k6 run --out=prometheus-remote-write \
  --server-url=http://localhost:9009/api/v1/push \
  script.js

The k6 run --out=prometheus-remote-write command sends test metrics directly to Mimir for analysis in Grafana.

#Example k6 Script

script.js - k6 scenario

import http from "k6/http";
import { check, sleep } from "k6";
 
export const options = {
  vus: 50,
  duration: "5m",
};
 
export default function () {
  const res = http.get("http://checkout:8080/health");
  check(res, { "status 200": (r) => r.status === 200 });
  sleep(1);
}

The vus: 50 configuration runs 50 virtual users for 5 minutes against the tested endpoint.

#Benchmarking the Observability Stack

#Dedicated Tools per Data Type

The LGTM Stack itself needs testing with the right tools:

• avalanche: tests Mimir's metric ingestion with large sample volumes.
• loggen: sends synthetic logs to Loki.
• trace generation tools: generate artificial traces for Tempo.
• Query performance testing: repeatedly tests the speed of heavy queries.

avalanche --remote-write-url http://localhost:9009/api/v1/push \
  --metric-count 1000 --series-count 100000

The avalanche command above simulates 1000 metrics with 100000 time series — testing how far Mimir's ingestion limits go.

#Measuring Query Performance

Beyond ingestion, queries need testing: run heavy queries repeatedly, record the duration, then compare before and after optimization. Grafana's Query inspector helps see each query's duration.

#Establishing a Performance Baseline

Before capacity testing, establish a clear baseline:

• Ingestion rates: how many samples per second can be processed.
• Query latencies: normal query response times at reasonable load.
• Resource utilization: CPU and memory usage of each component.
• Storage growth rates: how fast storage grows per data volume.

baseline:
  ingestion: 500k samples/second
  query_p95: 300ms
  storage: 40GB/day

This baseline documentation is what's used to compare later test results.

#Capacity Testing

#Three Main Scenarios

• Stress testing: load keeps rising until the stack fails — finding the maximum limit.
• Soak testing: reasonable load over a long period — finding memory leaks and gradual degradation.
• Spike testing: sudden surges — testing the ability to absorb traffic peaks.

stress: keep rising until failure
soak:  stable over a long period
spike: sudden surge then drops

The stress: keep rising until failure pattern determines your stack's scalability limit — important data for the capacity planning in episode 35.

#Closing

In episode 32 you understood load testing with k6 and remote write to Mimir, benchmarking the stack with avalanche, loggen, and trace generators, establishing performance baselines, and capacity testing with stress, soak, and spike scenarios.

The key takeaways:

• k6 sends test metrics directly to Mimir.
• avalanche, loggen, and trace generators test the LGTM backends.
• Performance baselines are the comparison point for all tests.
• Stress, soak, and spike answer different capacity questions.
• Test in a separate environment, not production.

In the next episode 33 we'll discuss security and compliance — TLS encryption, authentication and authorization, protecting PII data in logs, GDPR compliance, and vulnerability management in containers and dependencies. An observability stack stores a lot of sensitive data — protecting it is a priority.