Learn Observability with the LGTM Stack - Metrics with Mimir - Introduction & Architecture

#Introduction

The first LGTM component you'll dive into is Grafana Mimir — a Prometheus-compatible metrics storage backend designed for large scale. Mimir takes the role of the place where all your application and infrastructure metrics are stored long term.

This episode covers Mimir's component architecture, a comparison with Prometheus and when to use each, the remote write path, and metric types and cardinality considerations. Understand the concepts in this episode because the PromQL queries in episode 7 will run on top of Mimir.

#Grafana Mimir Overview

#Prometheus-Compatible TSDB

Mimir is a TSDB (Time Series Database) that speaks the Prometheus protocol. That means the entire Prometheus ecosystem — the PromQL query language, metric format, and remote write — works directly without modification. This makes migrating from Prometheus to Mimir very smooth.

The features that set Mimir apart from a standalone Prometheus:

• Horizontally scalable architecture: every component can be split across many instances, with no single point of failure.
• Long-term storage: data is stored in object storage like S3 rather than local disk, so retention can span years.
• Multi-tenancy support: one Mimir cluster serves many teams with data isolation — covered in episode 25.
• High availability design: built-in replication ensures data stays safe when an instance fails.

blocks_storage:
  backend: filesystem
  filesystem:
    dir: /data/blocks
  tsdb:
    dir: /data/tsdb
  bucket_store:
    sync_dir: /data/bucket-sync

The configuration above is the fuller version used in episode 4. In production, the backend: filesystem section is replaced with s3 or gcs — covered in episode 22.

#Mimir Architecture

Mimir splits the data path into small components, each of which can be scaled independently:

• Distributor: receives remote write data, validates it, and replicates to ingesters. It is the entry gate for all metrics.
• Ingester: stores data in memory for a short period and writes blocks to object storage. The most recent data is read from here.
• Querier: executes PromQL queries by combining ingester and object storage data.
• Query-frontend: receives queries from users, splits large queries, and caches results.
• Store-gateway: reads blocks from object storage and optimizes lookups.
• Compactor: merges small blocks into larger ones and applies retention.
• Ruler: evaluates recording and alerting rules — covered in episode 19.

remote write -> distributor -> ingester -> object storage
                         ^                        |
query -> query-frontend -> querier <- store-gateway

The distributor -> ingester -> object storage diagram is the core metric storage path. Meanwhile the query path goes through query-frontend, querier, and store-gateway, which reads from object storage.

#Prometheus vs Mimir

#When to Use Each

• Use Prometheus when: small scale, a single cluster, short retention, and simple operational needs. Prometheus is also great as an edge collector that scrapes targets in the field.
• Use Mimir when: many clusters or many teams, long retention, the need for global queries, and large scale. Mimir does not scrape; it receives remote write from Prometheus or agents.

#Migration Path and Compatibility Layer

A common migration path is to keep Prometheus as the scraper while adding remote write toward Mimir. That way, metrics remain available in Prometheus for short-term needs and are copied to Mimir for the long term.

remote_write:
  - url: http://mimir:9009/api/v1/push
    headers:
      X-Scope-OrgID: local

The remote_write block above goes in the Prometheus config. The X-Scope-OrgID header is required when multi-tenancy is enabled; in local mode its value is arbitrary.

#Metric Types and Cardinality

#Metric Types

Mimir understands the four Prometheus metric types:

• Counter: a monotonically increasing value, for example http_requests_total. It can only be reset.
• Gauge: a value that goes up and down, for example temperature_celsius.
• Histogram: a distribution of observations, storing cumulative buckets to compute percentiles.
• Summary: a distribution with percentiles computed client-side, suitable when aggregation isn't needed.

counter | gauge | histogram | summary

Remember this counter | gauge | histogram | summary pattern; the chosen type determines which query functions are valid, such as rate which only works for counters.

#The Danger of Cardinality

Cardinality is the number of unique label combinations on a metric. For example: a metric with a user_id label holding a thousand values will create a thousand time series. Multiply that by a few more labels and the count explodes, burdening storage and queries.

• Limit labels to values with a small, stable number of distinct options, such as method, status, or service.
• Avoid high-cardinality labels like user_id, email, or trace_id.
• Take advantage of features like Mimir's adaptive metrics to manage cardinality automatically — covered in episode 35.

#Closing

In episode 6 you understood that Mimir is a Prometheus-compatible TSDB with a modular, horizontally scalable architecture, learned the role of each component from distributor to ruler, learned when to use Prometheus or Mimir, understood the remote write path, and the metric types and the dangers of cardinality.

The key takeaways:

• Mimir is Prometheus-compatible and stores data in object storage.
• Split architecture: distributor, ingester, querier, store-gateway, compactor, ruler.
• Prometheus for small scale, Mimir for long retention and multi-team.
• Remote write is the migration bridge from Prometheus to Mimir.
• Four metric types: counter, gauge, histogram, summary.
• High cardinality is the main enemy of cost and performance.

In the next episode 7 we'll discuss PromQL — the Prometheus query language used to read all metrics in Mimir, from instant vectors and range vectors, aggregation operators, common query patterns like rate and histogram_quantile, to performance best practices. Your dashboards will start speaking the language of metrics.