Learn Observability with the LGTM Stack - OpenTelemetry Collector Deep Dive

#Introduction

In episode 3 you got a brief introduction to the OpenTelemetry Collector. Now it's time to dissect it deeper, because the Collector is the central point of your telemetry architecture — receiving data from applications, processing it, then distributing it to Mimir, Loki, and Tempo.

This episode covers the Collector architecture consisting of receivers, processors, exporters, and extensions, agent and gateway deployment modes, popular components for each part, and best practices for building reliable pipelines.

#Collector Architecture

#Four Main Blocks

• Receivers: receive telemetry from various sources, for example OTLP, Prometheus, or files.
• Processors: process data in the middle of the pipeline — batching, filtering, sampling, and enrichment.
• Exporters: send data to final destinations, such as Tempo, Mimir, or other backends.
• Extensions: supporting features like health checks and profiling, without participating in the data flow.

#Pipeline Configuration

Pipelines connect receivers, processors, and exporters in one flow:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  batch: {}
 
exporters:
  otlp:
    endpoint: tempo:4317
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

The service.pipelines.traces block defines the trace flow: from the otlp receiver, through the batch processor, to the exporter toward Tempo.

#Deployment Modes

#Agent Mode

In agent mode, the Collector runs on every node, receiving telemetry from local applications then sending it to a gateway or directly to the backend. Its advantage: it's close to the data source, making local scraping and relabeling easy.

#Gateway Mode

In gateway mode, a centralized Collector receives data from many agents or applications. Its advantage: one point for heavy processing, smart sampling, and organizational policies.

#Hybrid Deployment

A combination of the two is most common: an agent on each node for local collection, and a centralized gateway for aggregation and distribution to backends.

apps -> agent (per node) -> gateway (centralized) -> Tempo/Mimir/Loki

The agent (per node) -> gateway (centralized) pattern is the architecture you'll most often encounter in large-scale production.

#Popular Receivers and Processors

#Common Receivers

• OTLP receiver: receives traces, metrics, and logs via gRPC and HTTP.
• Prometheus receiver: performs Prometheus-style scraping from targets.
• Jaeger and Zipkin receivers: accept legacy trace formats.
• Filelog receiver: reads logs from files.
• Host metrics receiver: collects system metrics.

#Important Processors

• Batch: combines data into large batches for efficiency.
• Memory limiter: limits Collector memory usage.
• Attributes: adds, changes, or removes attributes.
• Resource: adds resource attributes like the cluster name.
• Tail sampling: sampling based on decisions after a trace completes.
• Filter: removes unnecessary data.

processors:
  batch:
    send_batch_size: 10000
    timeout: 10s
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

The limit_mib: 512 configuration keeps the Collector's memory under control below 512 MiB.

#Exporters and Best Practices

#Commonly Used Exporters

• OTLP exporter: sends data to other backends that support OTLP.
• Prometheus exporter: remote writes metrics to Mimir.
• Loki exporter: sends logs directly to Loki.
• Jaeger exporter: forwards traces to Jaeger if present.
• Debug exporter: prints data to the console for troubleshooting.

#Best Practices

• Design minimal pipelines: only install the processors you actually need.
• Optimize resources: enable the memory limiter and tune the batch size.
• Error handling: use retry and queue on exporters.
• High availability: run two gateway instances behind a load balancer.

#Closing

In episode 16 you understood the Collector architecture consisting of receivers, processors, exporters, and extensions, the agent, gateway, and hybrid deployment modes, popular components for each part, and best practices for building reliable and efficient pipelines.

The key takeaways:

• Pipelines connect receivers, processors, and exporters.
• Agent on each node, centralized gateway for heavy processing.
• Batch and memory limiter keep Collector performance in check.
• Filtering and sampling reduce data volume.
• Heavy processors go at the gateway, not the agent.

In the next episode 17 we'll discuss advanced Grafana Alloy features — a comparison with the OTel Collector, deeper into the River language, discovery and relabeling components, and clustering mode for high availability. The two collectors you know will connect into a single collection strategy.