Learn Observability with the LGTM Stack - Structured Logging Best Practices

#Introduction

Loki can store unstructured logs, but you'll struggle to extract meaning from them. Structured logging — writing logs as key-value pairs or JSON — is the practice that turns logs from raw text into data that can be queried, aggregated, and correlated.

This episode covers the principles of structured logging, what should and shouldn't be recorded, a comparison of logging libraries per language, and techniques for correlating logs with TraceID. The quality of the logs in episode 10 will be largely determined by the practices you learn now.

#Structured Logging Principles

#Key-Value and JSON

Structured logs always take the form of named fields. Each event becomes a single JSON object with a timestamp, level, and context:

{
  "ts": "2026-08-10T11:20:05Z",
  "level": "error",
  "service": "checkout",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "order_id": "ord-1042",
  "error": "payment gateway timeout",
  "retry_count": 3
}

Note the consistency of field names: ts, level, service, trace_id are used across all services so cross-service queries stay consistent.

#Choosing the Right Log Level

• DEBUG: detail for development, avoid in production.
• INFO: important, routine business events.
• WARN: abnormal conditions that haven't failed yet.
• ERROR: failures that need attention.

Excessive levels in production inflate log volume and storage costs without added value.

#What Should Be Logged

A few categories of events worth recording:

• Request and response data: method, path, status code, duration, and user agent.
• Error messages and stack traces: clear error messages along with the context of where they occurred.
• Business events: order created, payment succeeded, account created.
• Performance metrics: the duration of important slow steps.
• Security events: failed login attempts, suspicious access, permission changes.

Structured logging example with structlog

import structlog
 
logger = structlog.get_logger()
 
def create_order(order_id):
    logger.info("order.created", order_id=order_id, total=150000)

The logger.info("order.created", ...) call produces JSON logs with fields that can be queried directly in Loki.

#What Should Not Be Logged

#Sensitive Data

Never record:

• PII: email, phone numbers, addresses, user identity data.
• Credentials: tokens, passwords, API keys, session cookies.
• Payment data: card numbers and sensitive transaction details.

{
  "ts": "2026-08-10T11:20:05Z",
  "level": "info",
  "event": "login.success",
  "password_hash": "jangan-pernah",
  "card_number": "4111-1111-1111-1111"
}

Fields like password_hash above must never appear in logs. The impact of such data leaks will be discussed further in episode 33.

#High-Cardinality and Verbose Logs

• High-cardinality values: don't log unique IDs as labels — keep them as fields in the line content.
• Excessive debug logs: don't enable DEBUG in production except during targeted investigations.

#Logging Libraries per Language

• Go: zap and zerolog — both fast and support JSON output.
• Java: logback and log4j2 with JSON pattern encoders.
• Python: structlog and python-json-logger.
• Node.js: winston and pino — pino is famously very lightweight.

pip install structlog

After pip install structlog, you can directly use the create_order example above.

#Correlating Logs with TraceID

The greatest value of structured logging appears when logs are correlated with traces. Every service should inject the TraceID into every log line:

• Request IDs: a unique ID per request, can be generated at the gateway.
• Trace IDs: from the OpenTelemetry context — covered in episode 14.
• User IDs and session IDs: user context when relevant.
• Correlation headers: forwarding IDs between services via HTTP headers.

gateway -> orders -> payment -> loki
   trace_id="4bf92f..." unified across all services

With a consistent trace_id in all logs, you can trace the entire journey of a request with a single query in episode 18.

#Closing

In episode 12 you understood the principles of structured logging with JSON and consistent field names, what is and isn't worth logging, the best libraries per programming language, and the importance of correlating logs with TraceID and Request ID.

The key takeaways:

• Structured logs are key-value pairs a machine can query.
• Keep field names consistent: ts, level, service, trace_id.
• Never log PII, credentials, and payment data.
• Choose a JSON logging library that fits your team's language.
• TraceID connects logs to traces across all services.

In the next episode 13 we'll discuss distributed tracing with Tempo — an object-storage-based trace backend, its component architecture, span and sampling concepts, and a comparison with Jaeger. Now the third pillar of observability takes the stage.