Learn Quarkus - Observability & Production Support

#Introduction

Episodes 9 and 21 made your application healthy and deployed. But production is where the unexpected happens: latency spikes, occasional errors, one service complaining about another. To survive, you need production-grade observability — not just a health check.

Episode 22 covers advanced observability: distributed tracing with OpenTelemetry, centralized logging with correlation IDs, production-ready metrics, health, and alerting, as well as incident response, error tracing, and service troubleshooting.

#Distributed Tracing and OpenTelemetry Integration

#Adding the Extension

OpenTelemetry is the modern observability standard. Quarkus integrates with it natively:

./mvnw quarkus:add-extension \
    -Dextensions=opentelemetry,opentelemetry-exporter-otlp

#Exporter Configuration

Send traces to a collector or a backend like Jaeger, Tempo, or Grafana Cloud:

quarkus.opentelemetry.tracer.exporter.otlp.endpoint=http://collector:4317
quarkus.opentelemetry.tracer.sampler=on
quarkus.application.name=belajar-quarkus

quarkus.opentelemetry.tracer.exporter.otlp.endpoint points to the OpenTelemetry Collector, which forwards traces to the backend. With the on sampler, every request is traced.

#Automatic and Manual Tracing

HTTP requests are traced automatically. For custom spans inside a service:

Custom span

import io.opentelemetry.api.trace.Span;
import jakarta.enterprise.context.ApplicationScoped;
 
@ApplicationScoped
public class PaymentService {
 
    public void prosesPembayaran() {
        Span span = Span.current().makeCurrent();
        span.setAttribute("payment.method", "transfer");
        // process the payment
        span.end();
    }
}

Distributed tracing follows a single request across many services: each service adds a span, and the backend assembles them into one complete trace.

#Centralized Logging and Correlation IDs

#Correlation IDs

When a request crosses many services, you need to connect the logs from all of them. A correlation ID (or trace ID) links everything together. With OpenTelemetry, the trace ID is automatically available.

Add the trace ID to your logs via configuration:

quarkus.log.console.format=%d{yyyy-MM-dd HH:mm:ss} %-5p [%c] trace=%X{traceId} %s%e%n

%X{traceId} reads the MDC value for the trace ID. Now every log line carries the trace ID, and you can find all the logs belonging to a single request.

#Centralized Log Aggregation

Send logs to a centralized system like Loki, Elasticsearch, or CloudWatch. Structured JSON logging configuration (episode 9) makes parsing easier. An example of sending to Loki:

quarkus.log.handler.gelf.enabled=true
quarkus.log.handler.gelf.host=loki-gateway
quarkus.log.handler.gelf.port=12201

Or export logs through an agent like Promtail on the cluster side. The important thing: all logs from all services gather in one queryable place.

#Production-Ready Metrics, Health, and Alerting

#Metrics and Health in Production

Combine metrics (episode 9) and health (episode 9) as the foundation of monitoring:

curl http://localhost:8080/q/metrics
curl http://localhost:8080/q/health

Prometheus pulls /q/metrics periodically; Kubernetes uses /q/health/live and /q/health/ready for probes.

#Alerting Rules

Good alerts catch problems before users notice them:

groups:
- name: quarkus
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
        / sum(rate(http_server_requests_seconds_count[5m])) > 0.05
    for: 10m
    labels:
      severity: page
    annotations:
      summary: Error rate di atas 5% selama 10 menit

This rule fires an alert if 5xx errors exceed 5% for 10 minutes. Alerts are routed to PagerDuty, Slack, or email according to severity.

#Incident Response, Error Tracing, and Service Troubleshooting

#Incident Runbooks

When an alert fires, the response must be structured:

1. Recognize: check dashboards and alerts to confirm
2. Isolate: identify the service and time range
3. Trace: query traces by trace ID and related logs
4. Recover: rollback, restart, or scale
5. Learn: postmortem and long-term fixes

#Tracing Errors with Traces

When a user reports an error, grab the trace ID from the report (or find it via logs), then:

curl "http://jaeger/query?service=belajar-quarkus&operation=POST"

Use the trace to see which requests failed, which service was problematic, and the duration of each span. Query via the API curl "http://jaeger/query?service=belajar-quarkus" or through the Jaeger UI. The trace ID is the bridge between user reports and root causes.

#Troubleshooting Latency

If an endpoint is slow, the trace shows which span consumes the time — a slow database query, an external call, or a queue. The combination of trace + metrics + log (the three pillars of observability) gives a complete answer without guessing.

#Wrap-Up

Episode 22 prepares you for production: understanding distributed tracing with OpenTelemetry, centralized logging with correlation IDs, production-ready metrics, health, and alerting, as well as trace-based incident response and troubleshooting.

Key takeaways:

• OpenTelemetry traces a single request across many services.
• The trace ID connects the logs of all services for the same request.
• Centralized logging gathers all services' logs in one place.
• /q/metrics and /q/health are the foundation of production monitoring.
• Alerting catches problems before users notice them.
• Incident response must be structured: recognize, isolate, trace, recover.
• The trace ID bridges user reports with the root cause.

In episode 23, the final episode of the series, we'll cover stable modern features and future trends — the latest stable Quarkus features, modern Quarkus 3.x capabilities, the SmallRye, Camel, Kafka, gRPC, and Kubernetes ecosystem, as well as strategies to keep your Quarkus applications future-proof and cloud-native.