Learn Observability with the LGTM Stack - Incident Response & Postmortems

#Introduction

All your observability work — metrics, logs, traces, alerts — culminates in a single moment: when an incident happens. This is when the collected data and the correlations you've built determine how fast the system returns to normal.

This episode covers the incident handling flow: detection, triage, escalation, debugging with observability, the postmortem culture, and the metrics used to assess the incident handling process.

#Incident Detection

#Four Detection Sources

• Alert-based detection: the primary detection from the alert rules in episode 19.
• Anomaly detection: deviations from normal patterns detected automatically.
• User reports: users report before alerts fire — a sign there's still a gap.
• Monitoring dashboards: operators spot anomalies from monitoring panels.

alert -> dashboard -> anomaly -> user report

The alert -> dashboard -> anomaly -> user report pattern shows the quality order of detection — at its best when users are never the first to know.

#Incident Response Flow

#Triage and Severity

Every incident goes through a triage process:

• Triage process: quickly assessing the impact and scope of the incident.
• Severity assessment: classifications like SEV-1 for a total outage and SEV-2 for degradation.
• Escalation procedures: raising the incident to the right team or level.
• Communication protocols: regular status updates to stakeholders.
• War room setup: a virtual space for active collaboration during major incidents.

sev-1: total service outage, needs immediate response
sev-2: significant degradation, can wait
sev-3: minor problem, handled during work hours

The definition sev-1: total service outage must be agreed by the whole team so escalation decisions are fast.

#Debugging with Observability

#The Metric → Trace → Log Flow

This is where the golden triangle from episode 18 is practiced:

metric (what) -> trace (where) -> log (details) -> root cause

The metric (what) -> trace (where) -> log (details) pattern runs in order: check metrics to confirm the scope of the problem, open traces to find the problematic span, then follow logs for error details.

#Timeline Reconstruction

With correlated data, the incident timeline can be reconstructed: when did metrics first deviate, which service errored first, and whether the pattern followed the latest release. These answers are immediately available if the instrumentation in episodes 14 and 18 has been prepared.

#Postmortem Culture

#Blameless Postmortems

The purpose of a postmortem is learning, not finding a scapegoat. Its principles:

• Blameless: focus on systems and processes, not individuals.
• Timeline documentation: record the full chronology based on data.
• Root cause analysis: use methods like the 5 Whys and Fishbone.
• Action items tracking: every finding becomes an assigned action.
• Learning organization: results are shared across the whole team.

why down? -> why timeout? -> why overload? -> ...

The why down? -> why timeout? pattern traces the root cause in layers — stopping at the symptom only produces shallow fixes.

#Cultural Impact

Teams that build a healthy postmortem culture aren't afraid to report incidents. Conversely, a blame culture makes incidents hidden and root causes never found.

#Incident Metrics

The following metrics assess the effectiveness of the handling process:

• MTTD (Mean Time To Detection): the time from when the incident occurs until it's detected.
• MTTA (Mean Time To Acknowledge): the time from detection until the team acknowledges it.
• MTTR (Mean Time To Repair): the time until the system returns to normal.
• Incident frequency: how often incidents occur.
• Severity distribution: the spread of incidents per severity.

#Closing

In episode 21 you understood incident detection sources, the triage, severity, and escalation flow, debugging with the metric → trace → log flow, the blameless postmortem culture with the 5 Whys, and the MTTD, MTTA, and MTTR incident metrics.

The key takeaways:

• Alerts and dashboards must detect before users complain.
• Clear severity speeds up escalation decisions.
• The golden triangle works during incidents: metrics, traces, then logs.
• Blameless postmortems encourage a learning culture.
• MTTD and MTTR measure the impact of observability investment.

In the next episode 22 we'll discuss scaling Mimir for production — capacity planning, horizontal component scaling, high availability with replication and zone-awareness, performance tuning, and object storage configuration for production. Your simple local stack will start growing into an enterprise architecture.