Learn OpenClaw - Operational Readiness & Runbooks

#Introduction

Episode 20 closed with large-scale observability — you know how to see and watch OpenClaw from many angles. Episode 21 raises the level: from tools to people and processes. No matter how fast the system is, it's still run by people, and people need clear instructions when panicking. This is operational readiness.

Episode 21's roadmap: writing runbooks for policy incidents, defining ownership and support boundaries between teams, building routine audits and compliance reviews, and setting up on-call and incident management that doesn't burn out the team.

#Runbooks for Policy Incidents

A runbook is a step-by-step instruction set that can be followed when an alert fires — even by someone newly joined or panicking at 3 AM. A good runbook answers three questions: what happened, who should be involved, and what's the first safe step to take.

runbook:
  id: openclaw-policy-denial-spike
  severity: SEV2
  symptoms:
    - "PolicyDenialSpike alert is firing"
    - "5xx errors rising on specific services"
  checklist:
    - "Open the policy dashboard and check which policies changed in the last 24 hours"
    - "Check the last commits in the policy repo via the GitOps deployment logs"
    - "If a new change is found, git revert and sync"
    - "Watch the denial rate return to normal for 15 minutes"
    - "Categorize the incident and write a postmortem"
  contacts:
    - "Platform on-call: @platform-oncall"

Note the checklist order above: diagnose first, then act. Panic makes people immediately change policies — which actually makes things worse. A runbook forces steps to be checked one by one. Store runbooks somewhere anyone can access, for example a repo or wiki, and write them in language understandable without prior context.

A runbook isn't a dead document. After every incident, update it with what actually proved useful: the command that saved the day, the sequence that didn't work, or data that turned out to be misleading. An outdated runbook is as dangerous as having none — people place trust in instructions that are no longer relevant.

#Ownership and Support Boundaries

Policy incidents can come from two different directions: a broken control plane (the platform's business) or a wrongly written policy (the application team's business). Without clear boundaries, every incident falls to the platform team and everyone waits on each other. Defining ownership resolves that deadlock.

• The platform team owns the control plane, installation, upgrades, and OpenClaw's overall stability.
• Application teams own the policies in their own namespaces, the traffic behavior of their services, and routing experiments.
• The security team owns standards and cross-team policies, for example a global default deny policy.

Documentation as a contract. Write down where the boundaries are: who may change which policies, who holds access to the control plane, and how incidents are handed off. If two teams disagree on who owns a policy, that's a symptom of missing documentation, not a technical problem. A clear service ownership matrix prevents the "nobody feels it's their business" situation.

Support boundaries in CI/CD. The validation from episode 19 also protects these boundaries. A gate requiring approval from the service owner before its policy is changed prevents changes from passing silently, and each team is responsible for its own policies.

RACI as a complement. Besides who owns, also write who is responsible for executing, who is consulted, and who is only informed. A RACI matrix prevents incidents from stalling mid-way because "everyone agrees it's important, but nobody feels responsible". For OpenClaw, for example: the platform team is responsible for recovering the control plane, application teams are consulted when their policies are affected, and management is merely informed via reports.

#Routine Audits and Compliance Reviews

Security that's never checked rots slowly: unused policies, ever-looser permissions, exceptions forgotten to remove. Routine audits turn neglect into routine.

kubectl get servicepolicies -o json | jq -r '.items[] |
  select(.spec.references == null) | .metadata.name'
openclawctl policy report --format markdown > audit-report.md

An audit isn't just a scolding — it produces artifacts. The report contains a list of active policies, policies never used (recall episodes 10 and 11), who owns each, when last changed, and whether they conform to standards. Keep the report as compliance evidence — external auditors and internal reviews both need records they can point to.

Schedule audits periodically: a quarterly review of loose policies, an annual review of all policies. Every finding has an owner and a due date. For comprehensive coverage, run openclawctl policy audit --all-clusters — it triggers an audit across clusters in one go and can be part of a monthly pipeline.

Compliance frameworks like SOC 2 or ISO 27001 demand evidence: who changed what, when, and how the change was approved. An audit report produced routinely doubles as an artifact for auditors — without a running process, you'll be scrambling to collect evidence at the last minute. The habit of periodic audits is an investment that pays twice: security stays healthy and compliance is documented.

#On-Call and Incident Management

Runbooks and ownership are useless if no one picks up the page when an alert fires. Healthy on-call requires clear rotation, escalation procedures, and documented communication habits:

• Severity matrix — SEV1 (network control lost), SEV2 (partial impact), SEV3 (minor impact, handled during work hours). Each level has its own response time and escalation path.
• Communication channels — one incident channel for discussion, one status page for announcements, and don't let reports scatter into private chats.
• Game days — rehearse synthetic incidents periodically, just like the recovery game days in episode 18. A practiced runbook works far better than one only read.

Escalation also needs a written form, not just "contact someone more senior". The severity definitions below can be a starting point:

severity:
  SEV1:
    response: 15 minutes
    escalate: "incident commander"
    channel: "#incident-sev1"
  SEV2:
    response: 60 minutes
    escalate: "platform lead"
    channel: "#incident-sev2"
  SEV3:
    response: "business hours"
    escalate: "owning team"
    channel: "#openclaw-ops"

Make this escalation procedure part of on-call onboarding, not a document first read when a SEV1 fires. On-call is a role run with preparation, not courage.

Good incident management isn't about never failing, but about failing with a process: document what happened, what didn't work, and one improvement to make. Every well-managed incident makes the next system a little stronger.

#Wrap-Up

In episode 21 you completed the human and process side of OpenClaw: runbooks anyone can follow during panic, ownership and support boundaries that remove deadlocks between teams, routine audits producing compliance artifacts, and on-call with clear rotation and escalation.

Key takeaways:

• A good runbook is diagnosis first, action second — and followable without context.
• Clear ownership (platform, application, security) prevents incidents floating without an owner.
• Routine audits must produce a report that can be pointed to as compliance evidence.
• A severity matrix, incident channels, and game days keep on-call from burning out the team.
• Every incident closes with a postmortem and one concrete improvement.

In episode 22 — the final episode of this series — you summarize everything: Production Hardening & Best Practices. You'll put together a security hardening checklist, build policy governance and lifecycle management, and prepare long-term maintenance and upgrades. See you there!