Learn Observability with the LGTM Stack - OpenTelemetry Instrumentation for Traces

#Introduction

Tempo provides the storage, but traces don't appear without instrumentation. In this episode you'll learn to produce quality traces with OpenTelemetry: from practical auto-instrumentation, creating manual spans for special logic, to context propagation that unifies traces across services.

By the end of the episode you'll know how to build applications that produce complete traces with consistent names and attributes — the raw material for TraceQL queries in episode 15.

#Auto-Instrumentation

#Per Programming Language

Auto-instrumentation automatically handles popular frameworks without changing business code:

• Java: opentelemetry-javaagent.jar run with the -javaagent flag.
• Python: the opentelemetry-instrumentation-* packages and the opentelemetry-instrument command.
• Node.js: @opentelemetry/instrumentation with an auto-loader.
• .NET: the OpenTelemetry.AutoInstrumentation package.
• Go: there is no auto-instrumentation — everything is done manually.

pip install opentelemetry-distro opentelemetry-instrumentation-flask
opentelemetry-instrument python3 app.py

The opentelemetry-instrument python3 app.py command runs the application with HTTP tracing active without touching the application code.

#Manual Instrumentation

#Creating Spans

For logic not covered by auto-instrumentation, create spans manually:

app.py - manual span

from opentelemetry import trace
 
tracer = trace.get_tracer("orders")
 
def process(order_id):
    with tracer.start_as_current_span("orders.process") as span:
        span.set_attribute("order.id", order_id)
        result = charge(order_id)
        span.set_status(trace.Status(trace.StatusCode.OK))
        return result

The construct with tracer.start_as_current_span("orders.process") ensures the span is automatically closed and becomes a child of the active span.

#Attributes, Events, and Exceptions

• Attributes: key-value pairs that can be queried, for example order.id.
• Events: points in time within a span, good for marking checkpoints.
• Exceptions: record errors complete with stack traces.
• Span status: mark spans as OK or ERROR according to the operation result.

Recording events and exceptions

span.add_event("charge.attempt", {"attempt": 1})
try:
    charge(order_id)
except Exception as exc:
    span.record_exception(exc)
    span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc)))

The pattern span.record_exception(exc) stores the error inside the trace, making debugging in episode 21 much faster.

#Context Propagation

#W3C Trace Context

For one trace to connect across services, context must be forwarded via HTTP headers:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The traceparent header carries the version, TraceID, SpanID, and sampling flags. Auto-instrumentation usually handles context injection and extraction automatically.

#Propagators and Baggage

• Propagators: determine how context is sent — HTTP headers for web, gRPC metadata for RPC.
• Baggage: carries cross-service data without creating spans, for example a user ID for business context.

python3 -c "from opentelemetry import propagate; print(propagate.get_global_text_map_propagator())"

The python3 -c ... command displays the global propagator currently active in the Python process.

#Instrumentation Best Practices

#Naming and Conventions

• Span names: use the verb.noun format like orders.process and checkout.pay — not arbitrary function names.
• Attributes: follow OTel semantic conventions, for example http.method, http.route, db.system.
• Consistency: consistent names and attributes keep TraceQL queries in episode 15 meaningful.

#Error Handling and Performance

• Mark span status as ERROR when an operation fails; don't leave it as success.
• Limit the number and size of attributes to reduce overhead.
• Use sampling at high traffic so costs stay under control.

#Closing

In episode 14 you understood auto-instrumentation for Java, Python, Node.js, and .NET, how to create manual spans with attributes, events, exceptions, and status, W3C context propagation with propagators and baggage, and best practices for span naming and attributes.

The key takeaways:

• Go has no auto-instrumentation; everything is manual.
• Spans are created with the with context so they always close.
• TraceID is forwarded via the W3C traceparent header.
• Span names follow a consistent verb.noun format.
• Record exceptions and ERROR status inside spans.

In the next episode 15 we'll discuss TraceQL — Tempo's query language for finding traces by span attributes, span relationships, comparison and logic operators, up to advanced queries and metrics from traces. The traces you produce will soon be explorable with precision.