Learn Observability with the LGTM Stack - Cost Optimization Strategies

#Introduction

The more telemetry data there is, the bigger the storage, compute, and network costs. Without a strategy, the observability bill can quietly balloon — especially in the cloud-native era where data volumes are relentless.

This episode covers cost control across three data types: metrics, logs, and traces, plus infrastructure cost optimization. The goal is simple: keep observability complete, but keep costs under control.

#Metric Cost Optimization

#Control Cardinality

Cardinality is the biggest metric cost killer. A single metric with a user_id label can create millions of time series.

• Cardinality reduction: remove high-cardinality labels from metrics.
• Recording rules: heavy aggregations are computed once, not on every query.
• Retention tuning: lower retention for low-value metrics.
• Sampling non-critical metrics: collect secondary metrics at longer intervals.
• Remote write filtering: filter at the collector so useless data is never sent.

prometheus.remote_write "default" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}

Combine remote_write with filter components to block high-cardinality metrics right at the collector.

#The Adaptive Metrics Feature

Mimir provides adaptive metrics that automatically flag high-cardinality series for removal. This feature is discussed further in episode 35.

As a rough guide: ten moderately labeled metrics are cheaper than one high-cardinality metric. That's why you should review metric labels periodically and involve service owners before removing any label — a removed label means historical data loses that dimension. Team agreement is far better than unilateral removal.

#Log Cost Optimization

#Label and Volume Control

• Label cardinality control: limit log labels as discussed in episode 23.
• Log level filtering: don't send DEBUG to production; drop useless WARN.
• Sampling strategies: sample INFO logs, keep all ERROR.
• Retention policies: keep recent logs on fast storage, move or delete old logs.
• Compression tuning: make sure compression is active at the collector.
• Storage tiering: move old chunks to a cheaper storage class.

loki.process "sample_info" {
  stage.match {
    selector = "{level=\"info\"}"
    stage.sampling {
      rate = 0.1
    }
  }
}

The rule stage.sampling rate: 0.1 only stores 10 percent of info-level logs — the most effective way to cut log costs without losing error context.

#Trace Cost Optimization

#Intelligent Sampling

• Intelligent sampling: store valuable traces — errors, slow ones, or priority services.
• Tail-based sampling: sampling decisions after a trace completes (episode 24).
• Storage retention: lower retention for normal traces.
• Query optimization: avoid queries that load large blocks without need.

store: errors, slow, critical services
sample: normal traces at a ratio

The pattern store: errors, slow, critical services ensures observability value stays high while costs drop. Combining sampling at the collector and retention at the backend gives two layers of control over trace costs.

#Infrastructure Costs

#Right-Sizing and Storage

• Right-sizing components: don't over-provision; measure the actual load of each component.
• Object storage cost management: use cold storage classes for old blocks.
• Network egress optimization: avoid expensive cross-region data transfers.
• Spot instances: for stateless components like queriers and distributors.
• Reserved capacity: long-term commitments for stateful components.

The combination of proper sampling and storage tiering usually yields the biggest savings. Prioritize savings on the data type with the largest volume — for most teams, that's logs, not metrics or traces.

Start with a small audit: list the five metrics, five log streams, and five services with the biggest costs, then apply one optimization step for each. The results are more measurable than trying to optimize everything at once.

#Closing

In episode 26 you understood metric cost control strategies with cardinality and recording rules, log costs with filters, sampling, and retention, trace costs with intelligent sampling, and infrastructure cost optimization.

The key takeaways:

• High cardinality is the biggest metric cost.
• Sampling INFO logs cuts costs without losing errors.
• Tail-based sampling stores valuable traces.
• Right-sizing and storage tiering save infrastructure costs.
• Measure observability costs regularly.

In the next episode 27 we'll discuss observability in Kubernetes — the architecture for monitoring nodes, pods, and clusters, metric sources like cAdvisor and kube-state-metrics, service discovery and relabeling, pod log collection, and tracing with a service mesh. Your LGTM Stack will move up a class into the world of Kubernetes.