Learn Observability with the LGTM Stack - Security & Compliance

#Introduction

Observability needs as much data as possible — but some of that data is dangerous. Logs contain PII, metrics leak business details, traces reveal internal flows. An under-protected stack is like opening the door to an organization's most sensitive data.

This episode covers LGTM Stack security practices, protecting telemetry data with redaction, compliance requirements, and ongoing vulnerability management.

#Basic Security Practices

#Encryption and Authentication

• TLS/SSL encryption: encrypt all traffic — between agents and backends, and between backends and Grafana.
• Authentication mechanisms: enable authentication on every component, don't leave them exposed.
• Authorization policies: restrict access with least privilege (episode 25).
• Network policies: limit communication between components in Kubernetes.
• Secret management: store credentials in Vault or Kubernetes Secrets, not in files.

server:
  http_tls_config:
    cert_file: /etc/tls/server.crt
    key_file: /etc/tls/server.key

The http_tls_config block enables TLS on Mimir's HTTP endpoints — a mandatory practice in production.

#Telemetry Data Security

#PII in Logs

Logs are the most vulnerable source of PII. Redaction must be applied before data enters storage:

loki.process "redact" {
  stage.replace {
    source = "message"
    replace = "***"
    expression = "email=[\\w.]+@[\\w.]+"
  }
}

The stage.replace rule replaces email addresses with three asterisks before logs are sent to Loki — sensitive data never enters storage.

Beyond pipeline redaction, apply an allowlist at the application level: only the fields that are truly needed are allowed into logs. This approach is stricter than trying to remove patterns one by one, because new patterns often appear as features grow.

Review redaction rules regularly. New patterns like account numbers or new token versions can slip through unnoticed, and only routine testing with sample data can catch them.

#Sensitive Metric Labels and Traces

• Sensitive metric labels: never use sensitive values as labels.
• Trace data sanitization: sanitize span attributes that contain PII.
• Data retention policies: apply minimal retention according to needs and compliance.

{
  "span.name": "login.process",
  "user.email": "jangan-ini",
  "auth.token": "jangan-ini"
}

Attributes like user.email above must never be sent as span attributes — all of them are queryable and exposed.

#Compliance Requirements

#Audit and Access

• Audit logging: record who accessed what and when (episode 31).
• Access logs: log access to the observability components themselves.
• Retention compliance: store data according to regulator-mandated periods.
• Data sovereignty: make sure data stays in permitted regions.
• GDPR considerations: users' right to deletion and data minimization.

#Vulnerability Management

#An Ongoing Process

• Container image scanning: scan images for CVEs before deploying.
• Dependency updates: update libraries and charts regularly.
• CVE monitoring: monitor security announcements for LGTM components.
• Security patches: apply high-priority security patches.

trivy image grafana/grafana:latest

The trivy image grafana/grafana:latest command scans the image for known vulnerabilities — add it to the CI pipeline so images with CVEs are rejected. Schedule recurring scans, not just once at build time, because new CVEs keep appearing.

#Closing

In episode 33 you understood basic security practices with TLS and authentication, telemetry data protection with PII redaction and trace sanitization, audit and GDPR compliance requirements, and ongoing vulnerability management.

The key takeaways:

• Encrypt all traffic and enable authentication on every component.
• Redaction prevents PII from entering log storage.
• Sensitive trace attributes must never be sent.
• Compliance determines retention and data location.
• Scan images and monitor CVEs regularly.

In the next episode 34 we'll discuss continuous profiling with Grafana Pyroscope — the fourth pillar of observability, CPU and memory profile types, Pyroscope integration with LGTM, flame graphs, and the profile → trace → metric → log workflow. The deepest performance layer will be opened.