Learn Observability with the LGTM Stack - Logs with Loki - Introduction & Architecture

#Introduction

The second pillar of observability is logs — detailed records of exactly what happened. But collecting logs from thousands of services is no trivial task: volume can reach terabytes per day, and storage costs are often the biggest burden.

Grafana Loki came to answer that problem. Inspired by Prometheus' design, Loki is built as a log aggregation system that is horizontally scalable and cost-effective, with a storage approach completely different from Elasticsearch. This episode covers Loki's architecture, its comparison with Elasticsearch, and log formats and parsing strategies.

#Grafana Loki Overview

#The Index-Free Approach

Loki does not index the content of every log line. Instead, it only indexes labels — key-value pairs attached to each log stream. Line content is stored as-is in compressed chunks.

The benefits of this approach are enormous:

• Horizontally scalable: because there's no giant index, adding capacity is just a matter of adding instances.
• S3-compatible storage backend: chunks are stored in cheap object storage like S3, GCS, or Azure Blob.
• Cost-effective: storage costs are far lower than the full-text indexing approach.
• Inspired by Prometheus: the label model, query, and metric exposure follow the Prometheus style.

docker exec loki /usr/bin/loki --version

The docker exec loki /usr/bin/loki --version command shows the version of the binary inside the container — a small habit that's useful when verifying a deployment.

#Loki Architecture

Like Mimir, Loki splits the data path into small components:

• Distributor: receives logs from clients, validates them, and writes to ingesters.
• Ingester: stores recent logs in memory and writes chunks to object storage.
• Querier: runs LogQL queries by combining ingester and storage data.
• Query-frontend: splits and caches large queries so they can be answered quickly.
• Compactor: merges and cleans chunks, and applies retention.
• Index-gateway (optional): centralizes index access in large deployments.

client -> distributor -> ingester -> object storage
                          ^              |
query -> query-frontend -> querier <- index-gateway

The distributor -> ingester -> object storage diagram shows the write path, while the query path is assisted by the index-gateway for label lookups.

#Loki vs Elasticsearch

#Different Approaches

• Indexing: Elasticsearch indexes almost all log fields for full-text search, Loki only indexes labels and relies on querying chunk content.
• Cost: because it doesn't index content, Loki's storage costs are far lower — the main factor in choosing Loki.
• Query performance: Loki excels at simple queries and label-based analysis; Elasticsearch excels at complex full-text search in log content.
• Use case fit: Loki suits system and application observability; Elasticsearch remains strong for search engines and deep text analysis.

loki: label-first, cheap, good for observability
elasticsearch: full-text, expensive, good for search

The pattern above doesn't mean one is better — choose based on need. loki: label-first is the default choice in this series because of cost.

#Log Formats and Parsing

#Structured vs Unstructured

Structured logs are in JSON or logfmt form so machines can parse them easily:

{
  "level": "info",
  "ts": "2026-08-10T10:05:11Z",
  "service": "orders",
  "order_id": "ord-1042",
  "msg": "order created"
}

Unstructured logs are just free text. Loki accepts both, but structured logs are far easier to analyze with LogQL in episode 10.

#Label Strategy and Parsing

An important principle in Loki: labels are limited, parsing happens at query time. Just use a few stable labels like job, service, and level; the rest is parsed from log content using LogQL parsers like json, logfmt, or pattern.

labels:
  job: checkout
  service: payment
  level: error

The labels concept above is similar to Prometheus labels — always think about cardinality before adding a new label.

#Closing

In episode 9 you understood that Loki is an index-free log aggregator that only indexes labels, learned its architectural components from distributor to compactor, compared its approach with Elasticsearch, and understood the importance of structured logs and disciplined label strategy.

The key takeaways:

• Loki doesn't index log content, only labels.
• Log chunks are stored in object storage, making it cheap.
• Distributor, ingester, and querier are Loki's core components.
• Loki wins on cost, Elasticsearch wins on full-text search.
• Limit labels; parsing happens at query time, not ingest time.

In the next episode 10 we'll discuss LogQL — Loki's query language for selecting log streams, filtering lines, parsing content, up to computing metrics from logs like rate and count_over_time. Your stored logs will soon be quick to explore.