Learn Observability with the LGTM Stack - OpenTelemetry - The Vendor-Agnostic Observability Standard

#Introduction

In episode 2 you got to know the three pillars of observability. But there's a practical question that hasn't been answered yet: how do your applications generate that telemetry in a consistent way without being locked into a single vendor? The answer is OpenTelemetry, or OTel for short.

This episode discusses the history of OpenTelemetry, the structure of its main components, the types of signals it supports, and why it became the de facto standard for modern instrumentation. OTel will accompany almost every following episode, because all the sample applications in this series are instrumented using the OTel SDK.

#OpenTelemetry Fundamentals

#History of the OpenTracing and OpenCensus Merger

Before OTel existed, the instrumentation ecosystem was fragmented. OpenTracing championed a vendor-agnostic tracing API standard, while OpenCensus, initiated by Google, brought libraries for traces and metrics with unified data collection. Both projects had their own strengths but were split in two directions.

In 2019, the two were announced to merge into OpenTelemetry. The project was then recognized as a CNCF (Cloud Native Computing Foundation) project and grew into the second most active project in the CNCF after Kubernetes. Its main goal is simple: one open standard for generating, processing, and exporting telemetry.

#Specification vs Implementation

OpenTelemetry is conceptually divided into two layers:

• Specification: the abstract rules that describe the API, SDK, data model, and semantic conventions. The specification contains no running code.
• Implementation: the actual SDK written for each programming language.

This separation ensures consistent behavior across languages. Official SDKs are available for Go, Java, Python, JavaScript, .NET, Ruby, PHP, Rust, and more — you can instrument applications in any language.

pip show opentelemetry-sdk

The pip show opentelemetry-sdk command will display the metadata of the installed SDK. You'll recognize package names like this when you practice in episodes 8 and 14.

#OTel Components

#API and SDK

• API layer: the interface developers use to create spans, record metrics, and write logs. The API is stable and has minimal dependencies.
• SDK implementation: the real implementation of the API that handles batching, sampling, exporter configuration, and hooks. Applications use the SDK to activate the API.
• Instrumentation libraries: packages that automatically instrument popular frameworks such as HTTP clients, database drivers, and message queues.

#OpenTelemetry Collector and Exporters

The OpenTelemetry Collector is the component that receives, processes, and exports telemetry from many sources to many destinations. You'll dive deep into it in episode 16. Meanwhile, exporters are plugins that send data to the destination backend:

• OTLP exporter: the native OTel protocol understood directly by Tempo, Mimir, and Loki.
• Prometheus exporter: exposes metrics in Prometheus format to be scraped.
• Jaeger exporter: sends traces to a Jaeger backend if needed.

app -> OTel SDK -> OTLP -> Collector -> Tempo
                          -> Mimir
                          -> Loki

The flow above is the standard pattern: one Collector serving three backends at once. Note that Collector -> Tempo happens via OTLP, while going to Mimir and Loki can use remote write or dedicated exporters.

#Signal Types

OpenTelemetry defines several types of telemetry signals:

• Traces: the representation of a request's journey across services, built from spans and context propagation.
• Metrics: numeric measurements, including counter, gauge, histogram, and cumulative.
• Logs: time-based events; OTel provides a log API and also accepts logs from external sources.
• Baggage: key-value pairs sent along with the context for cross-service needs, for example a user ID for business logic purposes.
• Future signals: profiling and events are being developed as official signals — profiling is already used via Pyroscope in episode 34.

traces | metrics | logs | baggage

Baggage is distinguished from trace context because it carries business data, not just request identity. Remember the pattern traces | metrics | logs | baggage as the list of signals always in use.

#Why OpenTelemetry Matters

#Avoiding Vendor Lock-in

With OTel, you write instrumentation once and export to any backend — Prometheus, Jaeger, Datadog, or the LGTM Stack. If one day you switch backends, your application code doesn't need to change.

#Standardization and Auto-instrumentation

• Standardized instrumentation: OTel semantic conventions ensure span names and attributes are consistent across teams, making cross-service queries meaningful.
• Auto-instrumentation capabilities: many languages provide agents that instrument frameworks automatically without changing code — for example the Java agent and the Python auto-instrumentation used in episode 14.
• Ecosystem momentum: almost every observability vendor now supports OTLP, making OTel the universal language of observability.
• Future-proof telemetry collection: because OTel is the most active open-source project, your skill investment is safe in the long run.

#Closing

In episode 3 you understood that OpenTelemetry is the open-source telemetry standard born from the merger of OpenTracing and OpenCensus, got to know the separation of specification and SDK, the API, SDK, Collector, and exporters components, and the supported signals from traces to baggage.

The key takeaways:

• OTel is a CNCF project resulting from the OpenTracing and OpenCensus merger.
• The API is separate from the SDK, with an SDK per programming language.
• The Collector is the central point for receiving and distributing telemetry.
• OTLP is the native export protocol understood by LGTM backends.
• OTel avoids vendor lock-in and supports auto-instrumentation.
• Four main signals: traces, metrics, logs, and baggage.

In the next episode 4 we start practicing: setting up the development environment — running the entire LGTM Stack along with the OpenTelemetry Collector and Grafana Alloy via Docker Compose, connecting data sources, and sending the first telemetry from a sample application. Get your terminal ready, because from now on we'll be typing a lot of commands.