Learn Observability with the LGTM Stack - Grafana Alloy Advanced Features

#Introduction

In episode 11 you used Grafana Alloy to collect logs. But Alloy is far more than just a file reader — it's a general-purpose collector that handles metrics, logs, and traces all at once, with a programmable River configuration language.

This episode covers when to use Alloy and when to use the OpenTelemetry Collector, a deeper exploration of the River language, discovery and relabeling components for service discovery, and clustering mode that makes Alloy run with high availability.

#Alloy vs OpenTelemetry Collector

#When to Use Each

• Use Alloy when: you're already in the Grafana ecosystem, need easily programmable dynamic configuration, and want one binary for all signals.
• Use the OTel Collector when: you want full conformity with the OTel ecosystem, need the largest receiver and processor ecosystem, or your team is already used to OTel YAML configuration.
• Complementary usage: the two are often used together — Alloy as the edge collector, the OTel Collector as the centralized gateway.

alloy (edge) -> otel collector (gateway) -> backend

The alloy (edge) -> otel collector (gateway) pattern is the most common combination in mixed deployments.

#Migration Path

Old Grafana Agent configuration can be converted to Alloy with the alloy convert tool. On the OTel side, most components have equivalents, so migration is generally gradual.

#The River Language

#Basic Syntax and Components

River is a component-oriented language:

• Component blocks: component "name" "label" followed by attributes inside curly braces.
• Variables and expressions: supports typed values like lists, maps, and functions.
• Flow control: data flows are connected with the -> arrow operator.

endpoint = "http://loki:3100/loki/api/v1/push"
 
loki.write "default" {
  endpoint {
    url = endpoint
  }
  external_labels = {
    env = "dev",
  }
}

The expression endpoint = "..." shows that attributes can use variables — making Alloy configuration programmable like code.

#Advanced Components

• Discovery components: discovery.kubernetes, discovery.docker, and discovery.relabel for finding targets.
• Relabeling: rewriting scrape target labels.
• Remote write: sending metrics with prometheus.remote_write.
• OTLP receivers and exporters: otelcol.receiver.otlp and otelcol.exporter.otlp.

discovery.docker "nodes" {
  host = env("DOCKER_HOST")
}
 
discovery.relabel "keep_web" {
  targets = discovery.docker.nodes.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "web-.*"
    action        = "keep"
  }
}

The discovery.relabel rule filters targets to only containers whose names start with web- — a typical service discovery pattern in production.

#Alloy Clustering

#Distributed Mode

Alloy supports clustering for horizontal scaling:

• Distributed mode: multiple Alloy instances form a cluster and share targets.
• Target allocation: scrape targets are distributed across cluster nodes.
• High availability: if one node dies, targets are moved to another node.
• State sharing: cluster members share information via a gossip protocol.

cluster:
  enabled: true
  name: logs-cluster
  join_peers:
    - alloy-1:12345

The cluster.enabled: true configuration enables cluster mode; join_peers connects an instance to existing cluster members.

#Closing

In episode 17 you understood Alloy's position versus the OpenTelemetry Collector and the strategy for using them complementarily, delved into the River language with variables and expressions, discovery and relabeling components, and clustering mode for high availability and target distribution.

The key takeaways:

• Alloy for edge collection, the OTel Collector for gateways.
• River allows configuration to be programmed with variables.
• Discovery and relabeling automate service discovery.
• Clustering distributes targets and maintains availability.
• Combining the two collectors is often the production architecture.

In the next episode 18 we'll discuss correlation between pillars — linking metrics to traces via exemplars, logs to traces via TraceID, and traces back to logs, up to the unified debugging flow of alert → metric → trace → log. This is the main advantage of the LGTM Stack that sets it apart from a mere collection of tools.