Learning Rust - Observability, Logging, and Tracing

#Introduction

An application running in production is almost impossible to understand without observability: logging, tracing, and metrics. When an error happens at 3 a.m., structured logs and complete traces are the first tools that save you.

Episode 14 builds the three pillars of observability in Rust: structured logging with tracing, distributed tracing with tracing-opentelemetry, and metrics with Prometheus. By the end of the episode, you will be able to answer three questions: what happened, where it happened, and how often.

#Logging with Tracing

#The Concept: Events and Spans

tracing is a modern logging framework: it distinguishes events (a moment in time, for example "request completed") from spans (a period, for example "processing request id 42"). Spans can nest, so an entire request can be tied together in one context.

cat > src/main.rs <<'EOF'
use tracing::{debug, error, info, span, Level};
use tracing_subscriber;
 
fn proses_request(id: u64) {
    let span = span!(Level::INFO, "request", id);
    let _entered = span.enter();
    info!("memulai proses");
    debug!("membaca data dari cache");
    error!("timeout saat menghubungi upstream");
}
 
fn main() {
    tracing_subscriber::fmt()
        .with_max_level(Level::DEBUG)
        .init();
 
    proses_request(42);
}
EOF
cargo run

span!(Level::INFO, "request", id) opens a contextual span, and _entered closes it at the end of the scope. info!, debug!, and error! record events inside the span. cargo add tracing-subscriber --features fmt,json adds the subscriber that prints to the terminal.

#Subscribers and the JSON Format

The subscriber is the part that receives events and spans. For production, switch the formatter to JSON so logs can be parsed by collectors such as Loki or OpenSearch: tracing_subscriber::fmt().json().with_current_span(true).init() produces one JSON object per line with timestamp, level, fields, and the active span. tracing::info!(user = "budi", ...) inserts structured fields — not just concatenated text.

#Instrumenting Handlers with an Attribute

#The #[tracing::instrument] Macro

Adding a span to every function manually is tedious. The #[tracing::instrument] macro does it automatically:

cat > src/main.rs <<'EOF'
use tracing::instrument;
 
#[instrument]
fn hitung_biaya(items: u32, harga: f64) -> f64 {
    items as f64 * harga
}
 
fn main() {
    tracing_subscriber::fmt().init();
    let total = hitung_biaya(10, 5.5);
    println!("total: {}", total);
}
EOF
cargo run

#[instrument] creates a span that follows the function, records the parameters, and closes when the function finishes. For axum handlers, #[instrument(skip(state))] excludes large state from the logs. This is the fastest way to make an application traceable.

#Distributed Tracing with OpenTelemetry

#Connecting Services Together

In a microservices architecture, a single request passes through many services. Distributed tracing connects all the segments through a trace ID propagated between services. The standard ecosystem for this is OpenTelemetry.

cat > src/main.rs <<'EOF'
use opentelemetry::global::set_tracer_provider;
use opentelemetry_otlp::new_exporter;
 
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let exporter = new_exporter()
        .tonic()
        .with_endpoint("http://otel-collector:4317")
        .build_export_pipeline()?;
    let provider = exporter.setup();
    set_tracer_provider(provider);
 
    tracing_subscriber::registry()
        .with(tracing_opentelemetry::layer())
        .init();
 
    tracing::info!("layanan observability siap");
    Ok(())
}
EOF
cargo run

The OTLP exporter sends traces to an OpenTelemetry Collector, which then forwards them to a backend such as Jaeger or Tempo. With this, a single trace ID connects a request that crosses many services — the primary debugging resource.

#Metrics with Prometheus

#Counters and Histograms

Metrics are numbers: request counts, durations, error rates. Prometheus pulls these numbers over the HTTP endpoint the application exposes. Start with Registry and IntCounter:

cat > src/main.rs <<'EOF'
use prometheus::{Encoder, IntCounter, Opts, Registry, TextEncoder};
 
fn main() {
    let registry = Registry::new();
    let counter = IntCounter::new(
        Opts::new("http_request_total", "total request http"),
    )
    .unwrap();
    registry.register(counter).unwrap();
 
    let mut buffer = Vec::new();
    let encoder = TextEncoder::new();
    encoder.encode(&registry.gather(), &mut buffer).unwrap();
    println!("{}", String::from_utf8(buffer).unwrap());
}
EOF
cargo run

Registry holds all the metrics, and TextEncoder produces the format Prometheus understands. In a server application, attach a handler that returns this text at the /metrics endpoint, like the axum handler in episode 10 — Prometheus scrapes it periodically, and Grafana visualizes it. IntCounter counts events; for durations, use Histogram or Summary.

#Optional: OpenTelemetry Metrics

If your stack already uses OpenTelemetry, opentelemetry also provides a metrics API that can be exported via OTLP. The choice is consistent: use prometheus directly for a scrape endpoint, or opentelemetry for a unified pipeline with traces.

#Observability Practices

A few habits that make observability useful rather than just noisy:

• Log at the right level: error for failures, warn for degradation, info for lifecycle.
• Always include structured fields — do not concatenate text into the message.
• Create spans at service boundaries and let #[instrument] handle the rest.
• One trace ID per user-facing request, propagated to all downstream calls.
• Three required metrics: request rate, error rate, and latency percentiles.

#Closing

Key takeaways:

• tracing distinguishes events and spans; the subscriber determines format and destination.
• #[tracing::instrument] instruments functions automatically.
• JSON logging so collectors can parse the logs easily.
• OpenTelemetry connects traces between services with a trace ID.
• Prometheus scrapes metrics from the /metrics endpoint.
• Three required metrics: rate, error rate, and latency.

In the next episode 15 we will discuss resilience, graceful shutdown, and health checks — shutting down the server cleanly via signal handling, providing health checks and readiness probes for Kubernetes, as well as applying retry, timeout, and circuit breaker with tower and tower-http. Your service will be battle-tested.