Learn Observability with the LGTM Stack - Scaling Mimir for Production

#Introduction

In episode 6 you ran Mimir as a single binary for development. In production, the load is far heavier: millions of time series, thousands of queries per second, and high availability demands. This is where Mimir's modular architecture shows its purpose.

This episode covers the steps toward production Mimir: capacity planning, horizontal component scaling, high availability design, performance tuning, and the right object storage configuration.

#Capacity Planning

#Estimating Requirements

Before adding instances, first calculate the load to be handled:

• Metric cardinality estimation: number of time series = metrics times label combinations. Watch out for metrics with high-cardinality labels.
• Ingestion rate calculations: samples per second entering, using the formula time series divided by the scrape interval.
• Query load estimation: the number of queries per second and the range load on dashboards.
• Storage requirements: estimates based on sample count and size per sample.

samples/second = time-series / scrape-interval
storage/day  = samples/second * 24 * 3600 * sample-size

The formula samples/second = time-series / scrape-interval is the starting point for all Mimir capacity calculations.

#Horizontal Scaling

#Components to Scale

Each component scales according to its role:

• Distributor: stateless — easy to multiply; load is distributed via a load balancer.
• Ingester: stateful — data in memory; replication is managed with the replication factor.
• Querier: stateless — scaled out to handle query load.
• Store-gateway: extends caching and access to object storage blocks.

ingester:
  replication_factor: 3

The value replication_factor: 3 means every time series is written to three different ingesters — if one dies, data remains available.

#High Availability

#Zone-Aware Replication

Distributing replicas across different zones protects against a single-zone failure:

• Replication factor: at least 3 for tolerance of one failure.
• Zone-aware replication: replicas are spread across different zones.
• Consistency guarantees: Mimir uses a quorum mechanism when reading and writing.
• Failure scenarios: plan what happens when the distributor, ingester, or storage fails.

zone-aware-replication:
  enabled: true

Enable zone-aware-replication: true and distribute ingesters across at least three zones to protect against zone failures.

#Performance Tuning

#Ingestion and Query Optimization

• Ingestion optimization: enlarge batches and parallelize processing at the distributor.
• Query optimization: start queries with strict label selectors, use recording rules.
• Compaction tuning: adjust the compaction interval so the number of blocks stays reasonable.
• Caching strategies: use caching for the query-frontend and store-gateway.

#Object Storage Configuration

#S3/GCS/Azure Blob

For production, replace the filesystem with object storage:

blocks_storage:
  backend: s3
  s3:
    endpoint: s3.amazonaws.com
    bucket_name: mimir-blocks
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

The block s3: bucket_name: mimir-blocks moves block storage from local disk to S3 — a prerequisite for scaling and long retention.

#Lifecycle and Cost

• Lifecycle policies: set up transitions of old data blocks to cold storage.
• Cost optimization: choose storage classes according to access — hot for new, cold for old.
• Access patterns: rarely accessed blocks are only read again during old range queries.

#Closing

In episode 22 you understood capacity planning with sample and time series calculations, horizontal scaling of Mimir components, high availability design with replication and zone-awareness, performance tuning, and object storage configuration for production.

The key takeaways:

• Calculate capacity before adding instances.
• Stateful ingesters need replication; distributors and queriers are stateless.
• Replication factor 3 with zone-awareness protects against failures.
• Measure dashboards before adding resources.
• Object storage is a prerequisite for scaling and retention.

In the next episode 23 we'll discuss scaling Loki for production — monolithic, simple scalable, and microservices deployment modes, ingestion and query scaling strategies, TSDB and BoltDB index management, storage optimization, and healthy label design. Loki in production demands the same discipline as Mimir.