Learn OpenClaw - Observability at Scale

#Introduction

In episode 19 you made sure all policy changes flow through GitOps. Episode 20 answers the question that arises once OpenClaw grows: how do you see all policies across many clusters without getting lost? In episode 7 you built basic observability essentials. Now it's time to scale observability itself.

Episode 20's roadmap: scaling metrics and dashboards across many clusters, correlating policy events with service health, building alerting for abnormal behavior, and keeping log volume under control.

#Scaling Metrics and Dashboards

As clusters grow, OpenClaw metrics grow too — every policy, every service, every target adds to the label series. The biggest problem in observability at scale isn't volume, but cardinality: exploding label combinations. A metric with per-service and per-policy labels can produce millions of series that torture storage.

{
  "name": "openclaw_policy_decision_total",
  "labels": {
    "cluster": "prod-eu-1",
    "namespace": "billing",
    "policy": "payment-access-policy",
    "action": "DENY"
  },
  "value": 4217,
  "timestamp": 1764800000000
}

Control cardinality from the start. Don't put always-changing values (like user IDs or pod names) into labels. Aggregate on the exporter side, store only the labels you actually use for grouping, and separate heavily-labeled metrics from those needing detailed storage. Prometheus federation and remote write to long-term storage like Thanos or Mimir separate hot data for fast queries from cold data for long retention.

Dashboards for each audience. One giant dashboard helps no one. Build three layers: a platform dashboard (control plane health, policy sync), a per-team dashboard (that team's policies, denials, mTLS status), and an executive dashboard (SLOs and trends). Use annotations to mark when policies are deployed via GitOps — later, when a weird metric appears, you can immediately connect it to the last deployment.

One pattern to watch: don't copy dashboards from another environment blindly. Staging and production traffic have different baselines; a dashboard used for debugging in staging could hide problems in production. Always verify thresholds and time ranges against the target environment.

#Correlating Policy Events with Service Health

Standalone policy metrics don't mean much. Their value appears when correlated with service health: does a policy denial correlate with 5xx errors? Does a latency increase coincide with a rate limit change? The key to correlation is shared labels — namespace, service, cluster — and a trace identifier attached to the policy event.

{
  "event": "policy_denied",
  "service": "payment-service",
  "namespace": "billing",
  "traceId": "8f1a0c1e00000001",
  "reason": "rate_limit_exceeded",
  "statusCode": 429
}

By carrying traceId in every policy event, you can jump from a denial metric to the same trace, then to the logs. This is the three-pillars flow discussed in episode 14: metrics tell you there's a problem, traces show the request path, logs give the detail. When the denial rate rises together with a shrinking SLO error budget, you know it's not just a weird number — there's real user experience being impacted.

#Alerting on Abnormal Behavior

Good alerting isn't about many alerts, but alerts that only fire when action is genuinely needed. Start with a baseline: know the normal denial rate, GitOps sync lag, and mTLS error rate in each environment. Alerts are triggered by deviation from the baseline, not from arbitrary absolute numbers.

groups:
  - name: openclaw-policy.rules
    rules:
      - alert: PolicyDenialSpike
        expr: |
          rate(openclaw_policy_decision_total{action="DENY"}[5m])
          > 3 * rate(openclaw_policy_decision_total{action="DENY"}[5m] offset 1h)
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Denial spike on a policy"

The rule above compares the denial rate of the last five minutes with the denial rate one hour ago. A three-fold deviation sustained for ten minutes triggers the alert — this catches anomalies (an over-tight policy, a blocked service, or even an abuse attempt) without noise under normal conditions. You can use openclawctl alert list to see alert status from the OpenClaw side, then link it with Alertmanager for routing and deduplication.

#Managing Log Volume and Retention

Policy events at large scale can flood the log pipeline. Volume control starts at the source: OpenClaw logs are differentiated by importance — decision logs (every policy decision), audit logs (configuration changes), and runtime logs. They don't all need the same retention.

logging:
  sinks:
    - type: loki
      url: https://loki.internal/loki/api/v1/push
  include:
    - AUTH_EVENTS
    - CONFIG_CHANGE
    - POLICY_EVENTS
  sampling:
    POLICY_EVENTS: 0.1
  retentionDays:
    AUTH_EVENTS: 365
    POLICY_EVENTS: 30

Sampling POLICY_EVENTS at 10 percent controls volume without losing the picture, while AUTH_EVENTS and CONFIG_CHANGE are stored in full for audit and compliance needs. Remember the trade-off from episode 6: detail discarded during logging can never be recovered. A balanced sample is better than full logs that stop being sent because of overload.

Beyond that, make sure the log sink has buffer and retry. When a sink is full, failed-to-send logs should be dropped deliberately rather than hanging the whole pipeline; and monitor whether the sink is catching up through a lag metric. Logs delayed by days are worth almost nothing for an investigation.

#Wrap-Up

In episode 20 you scaled OpenClaw observability: controlling metric cardinality, building dashboards people actually use, correlating policy events with service health via traces and shared labels, building baseline-based alerts, and keeping log volume under control with sampling and tiered retention.

Key takeaways:

• Cardinality is the main enemy of large-scale metrics — control labels from the start.
• Observability value appears when metrics, traces, and logs are correlated, not standalone.
• Baseline-deviation alerts are far more useful than static thresholds.
• Dashboards should be built per audience; a giant dashboard is read by no one.
• Retention and sampling must be decided deliberately: discarded detail never comes back.

In episode 21, you prepare the team and its processes — Operational Readiness & Runbooks. You'll create runbooks for policy incidents, define ownership and support boundaries, and build routine audits and compliance reviews. See you there!