Learning Python - Observability, Monitoring & Maintenance

#Introduction

Once the application runs in production, the question changes: how do we know it's healthy? Episode 21 answers with observability — three pillars: metrics, logs, and traces. You'll learn to use Prometheus, OpenTelemetry, and Sentry.

We'll also cover performance budgets and alerting, so problems are detected before users complain. Observability isn't optional — it's the eyes and ears of a production application.

#Metrics with Prometheus

#Exposing Metrics

Prometheus collects metrics from applications via an HTTP endpoint. The prometheus-client library provides it:

pip install prometheus-client

Membuat metrik Counter

from prometheus_client import Counter, start_http_server
import random
import time
 
REQUEST_TOTAL = Counter("http_request_total", "Total request HTTP", ["method"])
 
start_http_server(8001)
 
for _ in range(5):
    REQUEST_TOTAL.labels(method="GET").inc()
    time.sleep(1)
    print("metrik dikirim")

Counter("http_request_total", "Total request HTTP", ["method"]) defines a counter metric with labels. start_http_server(8001) opens a metrics endpoint. Prometheus scrapes this endpoint periodically and stores the data.

#Tracing with OpenTelemetry

#Understanding Distributed Tracing

Tracing follows one request's journey through many services. OpenTelemetry is the standard for this:

pip install opentelemetry-api opentelemetry-sdk

Membuat span dengan OpenTelemetry

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
 
trace.set_tracer_provider(TracerProvider())
provider = trace.get_tracer_provider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
 
tracer = trace.get_tracer("belajar")
 
with tracer.start_as_current_span("proses-pembayaran"):
    print("membuat span pembayaran")

trace.get_tracer("belajar") gets a named tracer. tracer.start_as_current_span("proses-pembayaran") creates a span that measures the duration of an operation. Spans are exported to backends like Jaeger or Tempo for visualization.

#Structured Logging

#Structured Logs

Structured logs use JSON format so they're easy to analyze:

Logging terstruktur

import json
import logging
 
logger = logging.getLogger("aplikasi")
 
def proses(order_id):
    logger.info(json.dumps({
        "event": "order_processed",
        "order_id": order_id,
        "status": "success",
    }))
 
proses("order-123")

json.dumps({...}) converts the log dict into JSON. This format can be parsed by systems like Loki, Elasticsearch, and Cloud Logging. Structured logs are far easier to filter than free text.

#Error Monitoring with Sentry

#Integrating Sentry

Sentry collects errors and crashes automatically:

pip install sentry-sdk

Menginisialisasi Sentry

import sentry_sdk
 
sentry_sdk.init(
    dsn="https://contoh@o0.ingest.sentry.io/0",
    traces_sample_rate=1.0,
)
 
def panggil():
    raise ValueError("contoh error")
 
try:
    panggil()
except Exception:
    sentry_sdk.capture_exception()
    print("error dikirim ke Sentry")

sentry_sdk.init(dsn=...) connects the application to a Sentry project. sentry_sdk.capture_exception() sends the error along with its stack trace and context. Sentry also collects uncaught exceptions automatically in most frameworks.

#Performance Budgets and Alerting

#Setting a Performance Budget

A performance budget is an agreed performance limit — for instance, p95 response time under 300 ms. Metrics from Prometheus are used to measure it:

Mengukur latency histogram

from prometheus_client import Histogram
 
REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "Durasi request HTTP",
    buckets=(0.05, 0.1, 0.25, 0.5, 1, 2.5, 5),
)

Histogram("http_request_duration_seconds", ...) measures the distribution of request durations. Buckets determine the granularity of percentiles like p95. A histogram is the basis for measuring a performance budget quantitatively.

#Rule-Based Alerting

Alerts notify you when a metric violates a limit. An example Prometheus rule:

groups:
  - name: aplikasi
    rules:
      - alert: RequestLambat
        expr: histogram_quantile(0.95, http_request_duration_seconds_bucket) > 0.3
        for: 5m
        annotations:
          summary: "Response time p95 di atas 300ms"

histogram_quantile(0.95, ...) > 0.3 fires an alert when p95 exceeds 300 ms for 5 minutes. Alerts are sent to Slack, email, or PagerDuty. Good alerting limits noise — only notifying about genuinely important problems.

#Closing

Key takeaways:

• Prometheus collects metrics from the /metrics endpoint.
• OpenTelemetry traces requests across services with spans.
• Structured JSON logs make automated analysis easier.
• Sentry collects errors with context and stack traces.
• Performance budgets are measured with histograms and percentiles.
• Automated alerting notifies you when performance limits are breached.

In the next episode, episode 22, we'll cover migration, scaling teams, and governance — upgrading Python versions safely, deprecation policies and compatibility testing, monorepo versus polyrepo strategies, managing internal packages, plus coding standards and contribution guidelines for large teams. This is the closing of your Learn Python journey!