Learn Observability with the LGTM Stack - Instrumenting Applications for Metrics

#Introduction

In episode 6 you got to know Mimir and in episode 7 you learned to read metrics with PromQL. But a fundamental question hasn't been answered: where do metrics come from? The answer is instrumentation — the process of adding code to an application so it produces measurements.

This episode covers two instrumentation paths: using the Prometheus client library directly and using the OpenTelemetry SDK for metrics. You'll learn to create counters, gauges, and histograms manually, take advantage of auto-instrumentation, then expose the results so they can be scraped or exported via OTLP to Mimir.

#Instrumentation Libraries

#Two Main Paths

There are two complementary approaches:

• Prometheus client libraries: directly expose metrics in Prometheus format at an HTTP endpoint. Simple and directly connects to Mimir via scraping.
• OpenTelemetry SDK for metrics: generate metrics in the OTel model, then export to the Collector. More flexible because one code base can be sent anywhere.

The choice of language depends on your team. Official libraries are available for Go, Java, Python, and Node.js, for example prometheus_client for Python and prometheus-client for Go.

pip install prometheus-client

The pip install prometheus-client command installs the library used in the examples in this episode.

#When to Use Which

Use the Prometheus client when your stack is simple and Mimir is the only metrics destination. Use the OpenTelemetry SDK when you've already adopted OTel for traces and logs — one SDK handles all signals consistently.

#Manual Instrumentation

#Creating Counter and Gauge

The simplest example with prometheus_client:

metrics.py - counter and gauge

from prometheus_client import Counter, Gauge, Histogram, start_http_server
 
REQUESTS = Counter("http_requests_total", "Total HTTP requests", ["method"])
IN_FLIGHT = Gauge("http_requests_in_flight", "Requests sedang diproses")
LATENCY = Histogram("http_request_duration_seconds", "Latency request")
 
def handle(method):
    IN_FLIGHT.inc()
    with LATENCY.time():
        REQUESTS.labels(method=method).inc()
    IN_FLIGHT.dec()
 
start_http_server(8000)

Note the pattern start_http_server(8000) — this starts a small HTTP server that exposes metrics on port 8000.

#Naming Conventions and Labels

• Metric names end with _total for counters and _seconds for durations.
• Labels are only for low-cardinality, stable dimensions: method, status, service. Never use user_id or trace_id as labels.
• Use lowercase and underscores, following the Prometheus style.

#Auto-Instrumentation

Writing manual instrumentation for every framework is unrealistic. This is where auto-instrumentation helps: libraries automatically instrument HTTP servers, database clients, and message queues.

#Runtime and HTTP Metrics

With the OpenTelemetry SDK, runtime and HTTP metrics can be enabled with a few lines:

app.py - runtime and HTTP metrics

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor
 
reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint="http://localhost:4317"))
metrics.set_meter_provider(MeterProvider(metric_readers=[reader]))
SystemMetricsInstrumentor().instrument()

After running SystemMetricsInstrumentor().instrument(), system CPU and memory metrics start exporting to the Collector every period.

#Various Frameworks

The same concept applies in other languages: opentelemetry-instrumentation-express for Node.js, opentelemetry-instrumentation-jdbc for Java, and opentelemetry-instrumentation-net-http for Go. Each package adds spans and metrics for the target framework without changing business logic.

#Presenting Metrics to Mimir

#Pull Model with /metrics

Prometheus and Mimir pull metrics using the pull model: periodically contacting the application's /metrics endpoint.

curl -s http://localhost:8000/metrics

The output of curl -s http://localhost:8000/metrics is in Prometheus exposition format — lines of text with metric names, labels, and values.

#Push Model with Remote Write

The second path is push: the application or agent periodically sends data to Mimir's remote write endpoint. This is used when the application runs behind a network the scraper can't reach, or when using a collector like Alloy.

url: http://mimir:9009/api/v1/push
send_interval: 15s

The remote write mechanism is covered in more detail in episode 11 when you use Grafana Alloy as the collector.

#Service Discovery

When there are many services, endpoints can't be recorded manually. Prometheus and Mimir use service discovery — for example Kubernetes labels — to find scrape targets automatically. This approach is covered in depth in episode 27.

#Closing

In episode 8 you understood the two metric instrumentation paths: the Prometheus client library and the OpenTelemetry SDK, created counters, gauges, and histograms manually, took advantage of auto-instrumentation for runtime and HTTP metrics, and presented metrics via both pull and remote write.

The key takeaways:

• Instrumentation is the source of all metrics; without code, there is no data.
• Counter for events, gauge for values, histogram for distributions.
• Labels only for low and stable dimensions.
• Auto-instrumentation handles popular frameworks without changing logic.
• Pull uses a metrics endpoint, push uses remote write.

In the next episode 9 we'll discuss logs with Loki — the distributor, ingester, querier, and compactor architecture, the difference between the index-free approach and Elasticsearch, and log formats and parsing strategies. Metrics are now running; it's logs' turn to complete the second pillar.