Learn OpenClaw - Network Observability & Forensics

#Introduction

In episode 13 you built defenses: rate limit, traffic shaping, blocklist, and alerting. But after an attack is held off, the real work is just beginning — the questions that must be answered: who attacked, when it started, through which path, and whether any data escaped? The answers are in the data, and OpenClaw stores it.

Episode 14 enters the investigation domain: packet flow visibility and audit logs as the source of truth, incident investigation steps with OpenClaw logs, and combining metrics, traces, and logs to see an incident as a whole, not in pieces.

#Packet Flow Visibility

Before asking "why is the service down", you must be able to answer "what actually happened on the network". OpenClaw records packet flows — who talks to whom, which protocol, how many bytes, how long the connection lasted — in the form of flow logs. One flow log represents one conversation between two endpoints:

{
  "ts": "2026-08-03T10:41:12.000Z",
  "src": "10.244.3.21:53124",
  "dst": "10.244.1.8:443",
  "service": "web-store",
  "proto": "tcp",
  "action": "allow",
  "bytes_sent": 4821,
  "bytes_recv": 120934,
  "duration_ms": 342
}

Flow logs answer connectivity questions — whether a connection formed, through which port, and how large — without storing payloads. Enable flow recording per namespace or per service, and make sure log throughput is accounted for so buffers don't explode during an attack:

apiVersion: openclaw.io/v1
kind: FlowLogConfig
metadata:
  name: core-traffic
  namespace: openclaw-system
spec:
  namespaceSelector:
    matchLabels:
      tier: core
  samplingRate: 1.0
  capture:
    - connection_start
    - connection_end
    - policy_hit
  sink:
    endpoint: http://loki-observability:3100

samplingRate: 1.0 means every connection is recorded — a fitting number for critical namespaces. For high traffic, sampling at 0.1 or 0.01 keeps costs reasonable while still giving a picture of the patterns.

#Audit Logs as the Source of Truth

Unlike flow logs which record network facts, audit logs record decisions: which requests a policy denied, who tried to access a service without permission, which policy changed and by whom. Audit logs are append-only — they can't be altered after being written — so they can serve as evidence.

Each audit entry carries identity, action, resource, and decision:

{
  "id": "a-9182-k3l",
  "ts": "2026-08-03T10:41:12.000Z",
  "actor": {
    "type": "apiKey",
    "name": "ci-pipeline"
  },
  "action": "policy.create",
  "resource": "policies/deny-admin-egress",
  "decision": "allow",
  "traceId": "t-55fc-11ab",
  "cluster": "prod-eks"
}

What must always be recorded: who the actor is (actor), what action was taken (action), which object was touched (resource), and the outcome (decision). With a traceId, this entry can be connected to the upstream request trace — the bridge to the cross-system investigation we'll cover at the end.

#Investigating Incidents with OpenClaw Logs

When a DDoS alert fires, a disciplined investigation flow reduces recovery time. Open with the big question: when did the anomaly start? Look for deny patterns in the flow log with openclaw logs flow --since 6h:

openclaw logs flow --since 6h \
  --filter 'action=deny' \
  --sort bytes_sent
openclaw audit query --since 6h \
  --filter 'decision=deny' \
  --top actor

#Reconstructing the Timeline

After finding the start time, reconstruct the sequence of events. Combine flow logs, audit logs, and reject metrics to build the timeline: when the rate limit started rejecting, when the blocklist went active, and when the service returned to normal. This timeline becomes the skeleton of the incident report.

#Identifying Actors and Patterns

From the lines most frequent in --top actor, classify the actors: is it one leaked API key, a group of IPs from one region, or a distributed attack from thousands of IPs? Pay attention to temporal patterns — for example deny counts that only appear during one country's business hours — before deciding on a permanent block:

openclaw audit query --since 24h \
  --group-by srcIp,userAgent \
  --filter 'decision=deny' \
  --limit 20

#Combining Metrics, Traces, and Logs

Flow logs and audit logs alone aren't complete. Real incidents involve applications: a slowing database, timing-out requests, dwindling memory. This is where the three pillars of observability come together. Metrics answer what happened (error rate rising), traces answer where (which service is slow), logs answer why (which exception appeared).

The key is identity correlation — one shared traceId spread across all sources. When a request is denied by OpenClaw, use that trace id to pull the end-to-end trace:

openclaw trace get t-55fc-11ab
openclaw audit query --trace t-55fc-11ab

From there you see: the trace shows the request slowing at billing due to a database lock, the flow log shows thousands of unanswered SYN connections, and the audit log shows the deny-admin-egress policy was changed just 10 minutes before the anomaly. Three previously separate sources now become one complete story. Integrate OpenClaw metrics into Prometheus and Grafana dashboards so this correlation can be done in one place rather than jumping between tools:

openclaw metrics scrape-config
curl http://openclaw-control-plane:9091/metrics

#Wrap-Up

Episode 14 turned OpenClaw from merely a policy enforcer into an investigation tool: flow logs record network facts, audit logs store undeniable decisions, and the combination of metrics, traces, and logs gives a complete picture of an incident. When an incident happens, you no longer guess — you read.

Key takeaways:

• Flow logs answer connectivity; audit logs answer decisions — two different things that complement each other.
• Audit logs must be append-only and carry a traceId so they can be connected to other systems.
• Start an investigation with the timeline, then group denials by source to find patterns.
• Combine metrics, traces, and logs through one shared identity like traceId.
• Sampling helps control observability costs without losing the big picture.

The evidence is collected, but what if an incident happens because OpenClaw itself is slow? In episode 15 we polish the engine: Performance Tuning & Scalability — sizing control and data planes, optimizing policy evaluation, and horizontal scaling with high availability configuration. See you there!