Learn OpenClaw - Observability Essentials

#Introduction

In episode 6 you strengthened security: mTLS is active, service-to-service policies are running, and the audit trail records everything. But there's one problem — you believe the policies work, but you can't yet prove it. How many requests did the payment-access-policy deny? Which service is blocked most often? The answers to those questions live in the realm of observability.

This episode is OpenClaw's observation window. Its roadmap: first we recognize the types of metrics and logs OpenClaw produces, second we integrate with Prometheus for metric storage, third we build Grafana dashboards for visualization, and finally we dig into visualizing policy hits and traffic flows as input for decision-making.

#OpenClaw Metrics and Logs

#Four Types of Telemetry

OpenClaw generates telemetry that can be grouped into four categories: metrics as cumulative numbers like request counts and latency; logs as discrete events like policy decisions; traces that follow one request's journey across services; and access logs that record the details of each incoming and outgoing request. All four complement each other — metrics for trends, logs for detail, traces for the trail.

For now we focus on the two most fundamental: metrics and logs. Both are the easiest to set up and deliver value fastest. An example of important metrics always worth monitoring:

openclawctl telemetry metrics list
openclawctl telemetry logs tail --follow --since 1h

From the first command's output you'll see metric names like openclaw_policy_eval_total, openclaw_policy_deny_total, openclaw_mtls_handshake_failures, and openclaw_http_request_duration_seconds. These metrics are the raw material for all our analysis later.

#Controlling Cardinality

One thing you must understand from the start: metric labels are powerful but dangerous. Every new label combination (namespace, policy name, source, destination) forces Prometheus to store a separate time series. If labels are used without control, the number of metric series can explode — a condition called high cardinality — which overloads Prometheus and slows down queries.

apiVersion: openclaw.io/v1
kind: TelemetryConfig
metadata:
  name: cluster-telemetry
spec:
  metrics:
    enabled: true
    labels:
      - namespace
      - policyName
      - result
    excludedLabels:
      - clientIp
      - userAgent
  accessLogging:
    mode: LOG
    sampleRate: 100

By limiting labels to namespace, policyName, and result, you reject unique-character labels like clientIp and userAgent whose cardinality is unbounded. This is a design decision that saves your observability infrastructure in production.

#Integration with Prometheus

#Setting Up the Scraping Endpoint

Prometheus works on a pull model: it fetches metrics from the /metrics endpoint that OpenClaw exposes. So what you need is to make sure that endpoint is active and tell Prometheus its address. If you're using kube-prometheus-stack, you just add a ServiceMonitor — an object that tells Prometheus what to scrape.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: openclaw-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: openclaw
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Once the ServiceMonitor is applied, Prometheus automatically starts pulling metrics every 30 seconds. Verify via kubectl port-forward -n openclaw svc/openclaw-metrics 9090:9090 and open the Prometheus query UI, or check directly with the following commands:

kubectl get servicemonitor -n monitoring
kubectl port-forward -n openclaw svc/openclaw-metrics 9090:9090

If everything is healthy, you can run your first PromQL query: rate(openclaw_policy_deny_total[5m]) to see the deny rate per minute. This is a basic query you'll use often.

#Grafana Dashboards

#OpenClaw Dashboard Template

Metrics stored in Prometheus aren't useful until they're visualized. That's where Grafana comes in as your monitoring board. OpenClaw provides a dashboard template you can import directly into Grafana — usually a JSON file containing ready-to-use panels. You just import it via the Dashboards then Import menu, or through a provisioning file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasource
  namespace: monitoring
  labels:
    grafana_datasource: "1"
data:
  datasource.yaml: |
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-operated.monitoring:9090
        access: proxy
        isDefault: true

#Panels That Must Exist

A good dashboard doesn't show everything — it shows what matters. At minimum there are four panels your OpenClaw dashboard must have: total policy evaluations per second, deny rate per policy, mTLS handshake failures (an indication of certificate problems), and the request duration histogram for p95 latency. These four answer daily operational questions: are policies burdening the system, and is anything strange happening on the network.

# Deny rate per policy
sum(rate(openclaw_policy_deny_total{namespace="billing"}[5m])) by (policyName)
 
# p95 latency of allowed requests
histogram_quantile(0.95,
  sum(rate(openclaw_http_request_duration_seconds_bucket[5m])) by (le))

The two PromQL queries above are a solid starting point. The first panel shows deny per policy so you immediately know which policy denies the most; the second tells you how fast allowed requests run.

#Visualizing Policy Hits and Traffic Flows

#Policy Hits Dashboard

Policy hits are the count of how often a policy was evaluated and what the result was — ALLOW, DENY, or SKIP. Visualizing them as a stacked bar chart per policy lets you see each policy's load and anomalies at once: a policy suddenly spiking in deny count? It could be an attack, or it could be an application bug.

sum(rate(openclaw_policy_eval_total[5m])) by (policyName, result)

This query produces a series for each policy and evaluation result combination. Add a namespace label selector to narrow the scope: {namespace="billing"}. From there you can build a time series panel to see deny rate rise and fall throughout the day.

#Visualizing Traffic Flows

Metrics can tell you there's a problem, but not necessarily where. For that you need to see traffic flows — who talks to whom, through which ports, and with what result. Grafana can visualize this as a node graph: each service becomes a node, and each line between nodes represents traffic with thickness proportional to volume.

openclawctl observability flows --namespace billing --output json

The output of the command above contains a list of source-destination pairs with their volumes and statuses. The same data can be fed into a Grafana node graph panel. In a single glance, you immediately see which service talks the most, and whether there are traffic paths that shouldn't exist — for example billing suddenly accessing the storage service when no policy allows it.

#Wrap-Up

In episode 7 you opened OpenClaw's eyes: recognizing four types of telemetry with a focus on metrics and logs, controlling label cardinality so you don't wreck Prometheus, integrating a ServiceMonitor for scraping, building Grafana dashboards from template to custom panels, and visualizing policy hits and traffic flows for anomaly detection.

Key takeaways:

• Metrics, logs, traces, and access logs complement each other; start with metrics and logs.
• Keep label cardinality in check — never put unique-character labels like client IPs into metrics.
• The ServiceMonitor is OpenClaw's bridge to Prometheus; make sure the /metrics endpoint is always scrapeable.
• Dashboards must include policy evaluations, deny rate, mTLS failures, and p95 latency.
• A traffic flow graph helps find suspicious communication paths that aren't visible in aggregate numbers.

In the next episode, episode 8, we enter Phase 3 — advanced policy authoring. You'll write reusable policy templates, create parameterized rules with policy inheritance, and use conditionals and service labels in rules. See you there!