Learn Observability with the LGTM Stack - Continuous Profiling (Grafana Pyroscope)

#Introduction

Metrics show a request slowing down, traces show which span is slow, and logs show errors — but none of them explains why the CPU spends time in a particular function. The answer lies in continuous profiling.

Profiling is considered the fourth pillar of observability. This episode covers the profiling concept, profile types, Grafana Pyroscope as a continuous profiling platform, flame graphs, and the workflow for correlating profiles with metrics, traces, and logs.

#Profiling Introduction

#The Fourth Pillar of Observability

Profiling periodically samples program execution and reconstructs where time is spent:

• CPU profiling: which functions use the most CPU.
• Memory profiling: where memory allocation happens.
• Goroutine profiling: the status and count of goroutines (Go).
• Mutex profiling: lock contention that slows down concurrency.

metrics: what is happening?
logs:    what exactly is happening?
traces:  where is the problem?
profile: how is the code running?

The pattern profile: how is the code running? completes the previous three questions into full observability.

#Grafana Pyroscope

#Continuous Profiling Platform

Pyroscope is a profiling backend integrated with the Grafana ecosystem:

• Continuous profiling platform: profile samples are collected continuously, not occasionally.
• Integration with the LGTM stack: profiles can be correlated with metrics, logs, and traces.
• Flame graphs: visualizing resource usage hierarchy in a single picture.
• Differential profiling: comparing two profiles to find differences.

docker run -d --name pyroscope -p 4040:4040 \
  grafana/pyroscope:latest

The docker run -d --name pyroscope command runs the Pyroscope server on port 4040.

#Sending Profiles from Applications

Pyroscope integration in Python

import pyroscope
 
pyroscope.configure(
    application_name="checkout.service",
    server_address="http://localhost:4040",
)

The pyroscope.configure call connects the application to the profiling server — sampling starts automatically.

#Profiling Use Cases

#Four Main Scenarios

• Performance optimization: finding the hot path that consumes the most resources.
• Memory leak detection: seeing allocations that are never freed.
• CPU hotspot identification: functions that monopolize CPU and throttle throughput.
• Resource consumption analysis: understanding resource requirements per service.

#Profile Correlation Workflow

#Profile → Trace → Metric → Log

The performance debugging workflow now becomes four layers:

profile -> trace -> metric -> log

The pattern profile -> trace -> metric -> log reads: start from the flame graph showing the hot spot, open the trace to see the related span, compare with metrics to confirm the impact, then check logs for error context.

#Practical Workflow

1. A metrics dashboard shows p95 latency increasing.
2. TraceQL finds the slowest span.
3. The flame graph shows the function consuming the most time.
4. Logs provide the error context that triggered it.

This combination changes performance debugging from guessing into targeted investigation.

#Closing

In episode 34 you understood profiling as the fourth pillar of observability, CPU, memory, goroutine, and mutex profile types, Grafana Pyroscope with flame graphs and differential profiling, and the profile → trace → metric → log correlation workflow.

The key takeaways:

• Profiling answers how the code actually runs.
• CPU and memory profiling are the most common types.
• Pyroscope integrates natively with the Grafana ecosystem.
• Flame graphs show hot spots in a single picture.
• Profile correlation speeds up performance debugging.

In the next episode 35 we'll discuss the production deployment checklist — pre-production checks, operational practices, monitoring the monitoring itself, common pitfalls, and a modern feature checklist. All the knowledge from the previous 34 episodes will be summarized into one go-live guide.