Learn Observability with the LGTM Stack - Scaling Loki for Production

#Introduction

Loki running in a single process is enough for development, but production demands more: logs from thousands of services, terabyte-scale growth, and queries that must stay fast. Loki provides several deployment modes for every stage of growth.

This episode covers Loki's deployment modes, ingestion and query scaling strategies, index management, storage optimization, and label design — the factor that most determines long-term cost and performance.

#Loki Deployment Modes

#From Monolithic to Microservices

• Monolithic mode: all components in one process — enough for moderate load.
• Simple scalable mode: three targets (read, write, backend) that can be scaled — the most commonly used middle ground.
• Microservices mode: every component is separate — for the largest scale.
• Read/write split mode: separates the read and write paths.

monolithic -> simple scalable -> microservices

The monolithic -> simple scalable -> microservices pattern is the common growth path: upgrade the mode as needs grow, not from the start.

#Simple Scalable Mode

This mode splits into read and write targets:

target: read,write

The value target: read,write separates ingesters (write) from queriers (read), each scalable independently.

#Scaling Strategies

#Ingestion and Query

• Ingestion rate limits: set per-tenant limits so one tenant can't flood the cluster.
• Query performance: scale the read target when queries slow down.
• Retention policies: set how long logs are stored and old data is cleaned up.
• Compaction configuration: schedule compaction to merge chunks and remove expired data.

limits_config:
  per_stream_rate_limit: 3MB
  per_stream_rate_limit_burst: 15MB

The limit per_stream_rate_limit: 3MB protects the cluster from sudden surges in log flow.

#Index Management

#TSDB vs BoltDB

Loki has two index formats:

• TSDB index: the newest recommended format — built-in sharding and more efficient.
• BoltDB index: the legacy format — still supported but migration is recommended.

schema_config:
  configs:
    - from: "2026-01-01"
      index:
        period: 24h
      object_store: s3
      schema: v13

The schema_config block determines the schema format and index period — make sure to use the latest schema that supports TSDB.

#Index Caching

Frequently accessed indexes are cached to speed up queries. Caching can use memcached or redis in large deployments.

#Storage Optimization

#Chunks and Compression

• Chunk encoding: chunk size affects query efficiency — too small is expensive, too large is slow.
• Compression algorithms: gzip and snappy to reduce size.
• Retention vs cost: the longer the retention, the bigger the storage cost.
• Tiered storage: move old logs to a cheaper storage class.

compactor:
  retention_enabled: true
limits_config:
  retention_period: 30d

The value retention_period: 30d sets logs to be stored for 30 days — adjust to your needs and compliance in episode 33.

#Label Design

#Principles and Anti-Patterns

Labels determine cardinality — and cardinality determines cost:

• Cardinality management: limit the number of values per label.
• Static vs dynamic labels: static labels like job and cluster are safe; dynamic labels are dangerous.
• Anti-patterns to avoid: never use user_id, trace_id, or IP as labels.

safe: job, service, level, cluster, environment
dangerous: user_id, trace_id, ip_address, request_path

The list safe: job, service, level above is a starting point for healthy label design.

#Closing

In episode 23 you understood Loki's deployment modes from monolithic to microservices, ingestion and query scaling strategies with limits and retention, TSDB and BoltDB index management, storage optimization with compression and tiering, and healthy label design.

The key takeaways:

• Upgrade the deployment mode according to the growth stage.
• Limit per-stream ingestion rates to protect the cluster.
• Use the TSDB index, not BoltDB.
• Retention and tiering control storage costs.
• High-cardinality dynamic labels are an anti-pattern.

In the next episode 24 we'll discuss scaling Tempo for production — single binary versus microservices deployment, ingestion strategies, query optimization with bloom filters and caching, sampling strategies, and metrics derived from traces. Tempo will grow from a simple trace backend into a production service.