Learn Observability with the LGTM Stack - Collecting Logs with Grafana Alloy

#Introduction

Logs don't flow to Loki by themselves. Between the application and the backend you need a collector — a process that reads logs from various sources, processes them, then sends them. In this series, that role is held by Grafana Alloy.

Alloy is the successor of the Grafana Agent, designed as a vendor-neutral collector fully compatible with OpenTelemetry. This episode covers its architecture, the River configuration language, collecting logs from files and Docker, the processing flow, and sending logs to Loki.

#Grafana Alloy Introduction

#Why Alloy

Alloy combines the capabilities of the OpenTelemetry Collector, the Prometheus agent, and the Loki client in a single binary. Its key features:

• Vendor-neutral: can send data to LGTM, Datadog, or other backends.
• OpenTelemetry compatibility: accepts and processes OTLP signals.
• Dynamic configuration: configuration can be reloaded and reprogrammed at runtime.
• Programmable pipelines: data flows are built as a chain of components.

docker exec alloy alloy convert --version

The docker exec alloy alloy convert --version command helps verify the binary inside the container.

#Alloy Architecture

#Component-Based Design

Everything Alloy does is represented as a component. Each component receives data, processes it, and exposes results that can be connected to other components — forming a pipeline.

• Component-based design: every function is a small component that can be combined.
• Pipeline composition: one component's output becomes the next component's input.
• Data flow model: data flows from receivers to exporters through a chain of components.
• Configuration language (River): all pipelines are written in the River language.

loki.source.file "app_logs" {
  targets = [{
    __path__ = "/var/log/app/*.log"
  }]
}
 
loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}
 
loki.source.file.app_logs -> loki.write.default

Note the last line loki.source.file.app_logs -> loki.write.default — the arrow operator connects the source component to the destination component.

#Log Collection Methods

#File Tailing and Docker Logs

Alloy reads logs from various sources with different components:

discovery.docker "containers" {
  host = env("DOCKER_HOST")
}
 
loki.source.docker "container_logs" {
  host = env("DOCKER_HOST")
  targets = discovery.docker.containers.targets
}

loki.source.docker monitors Docker containers directly, while discovery.docker finds running containers.

#Kubernetes, Syslog, and Journal

Other available methods:

• Kubernetes pod logs: loki.source.kubernetes monitors pod logs.
• Syslog receiver: loki.source.syslog receives logs from network devices.
• Journal logs: loki.source.journal reads systemd logs.

file | docker | kubernetes | syslog | journal

The file | docker | kubernetes | syslog | journal pattern covers almost all common log collection needs.

#Log Processing

#Labels and Parsing

Before being sent, logs can be processed: extracting labels, parsing content, filtering, and adding metadata. The loki.process component loads a chain of stages:

loki.process "enrich" {
  stage.regex {
    source      = "message"
    expression  = "level=(?P<level>\\w+)"
  }
  stage.labels {
    values = {
      level = "",
    }
  }
}
 
loki.source.file.app_logs -> loki.process.enrich
loki.process.enrich -> loki.write.default

The stage.labels stage maps parsed fields into log labels — an important pattern for cardinality control.

#Filtering and Multi-Tenancy Routing

Unnecessary lines can be dropped in the pipeline. For multi-tenancy, Alloy can add a tenant header on write components — covered further in episode 25.

#Sending to Loki

#Remote Write and Batching

Sending to Loki uses the push protocol with batching, compression, retry, and backpressure configuration:

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
    batch_max_size = 1 << 20
  }
  external_labels = {
    cluster = "local",
  }
}

The batch_max_size value limits batch size so delivery is efficient, while external_labels adds labels at the collector side.

#Closing

In episode 11 you understood Grafana Alloy's role as the vendor-neutral collector succeeding the Grafana Agent, learned its component architecture and the River language, studied log collection methods from files, Docker, Kubernetes, syslog, and journal, and the processing and delivery flow to Loki.

The key takeaways:

• Alloy builds pipelines from chains of components connected by arrows.
• River is Alloy's configuration language.
• Log collection covers files, Docker, Kubernetes, syslog, and journal.
• Log processing extracts labels, parses, filters, and enriches.
• Batching and compression reduce delivery costs to Loki.

In the next episode 12 we'll discuss structured logging best practices — key-value and JSON principles, what should and shouldn't be logged, logging library comparisons per language, and correlating logs with TraceID. The data flowing to Loki is only as good as the quality of the logs your applications produce.