Learn Hermes AI Agent - Ops & Governance

#Introduction

Episode 20 gave you all the signals to see what is happening in the agent system. Now comes the harder question: what do you do when those signals point to a problem, and who has the authority to make the decision? In episode 21 we move from "monitoring" to "operating" — the day-to-day business of the team responsible for agents already in production.

This topic is not about smarter code, but about more disciplined processes: incident playbooks, rollback and safe mode mechanisms, governance policies, and responsible AI documentation. Here is the roadmap for this episode:

• Operational playbooks: incident response, rollback, and safe mode.
• Governance policies for agent usage and deployment.
• Documentation of responsible AI practices.
• Building a healthy on-call culture for agents.

#Incident Playbook: When the Agent Misbehaves

The observability in episode 20 tells you that something is wrong. What distinguishes a professional team from an amateur one is what happens afterwards. Agent incidents usually fall into three categories, and each has its own first response:

The key to a good playbook is a certain sequence of actions in the middle of panic. When the alarm goes off in the middle of the night, the mind does not work at its best — so write the steps down beforehand. Here is a short example playbook for an agent incident:

1. Stabilize: enable safe mode or roll back to a healthy release. Do not debug before the system is safe.
2. Isolate: turn off the involved tools, limit traffic to the problematic profile.
3. Investigate: collect the traces of the failed conversations, compare with the baseline.
4. Fix: release a small fix, test it on canary, do not push to all traffic at once.
5. Learn: write a postmortem, add the alert or test that prevents recurrence.

#Rollback and Safe Mode as Safety Nets

The two mechanisms from episodes 19 and 20 are now official operational tools: rollback restores behavior to a healthy release, and safe mode lowers risk without actually shutting down the service. Safe mode is an emergency mode that turns the agent conservative: only read-only tools are allowed, all risky decisions are escalated to a human, and certain answers are preferred over creative ones.

mode: safe
tools:
  allowed:
    - search-readonly
    - db-readonly
    - escalate-human
  denied:
    - execute-command
    - send-email
    - delete-record
behavior:
  allow_creative: false
  escalate_on_low_confidence: true
  max_attempts: 1

Enable safe mode as quickly as possible during an incident, while the team decides between a full rollback or a gradual fix:

hermes mode set --env production --safe

Keep in mind that safe mode is a safety net, not a permanent home. Set a time rule: safe mode must be reviewed within a maximum of a few hours, and the team must have a clear path out — whether rollback, fix, or an official decision to decommission the service.

#Governance Policies for Usage and Deployment

Governance is a set of rules about who may do what. Without governance, anyone could change the production prompt, add a dangerous tool, or deploy an untested model. Hermes provides a policy layer that checks every operation — from deployment to tool calls — against the rules the team has agreed upon.

apiVersion: policy.hermes.dev/v1
kind: DeployPolicy
spec:
  allowedRoles:
    - deploy-engineer
    - release-manager
  requireEvalScore: 0.85
  requireApprover: true
  environments:
    production:
      requireCanary: true
      minCanaryPercent: 10
    staging:
      requireCanary: false

The rules above require three things to deploy to production: an authorized role, a minimum evaluation score, and human approval. Important note: governance must be recorded and audited. Every denied or approved operation leaves a trail — who, when, and why. This trail later becomes evidence in the event of a dispute or investigation.

#Documenting Responsible AI Practices

The last pillar of governance is documentation. Good code without decision documentation makes the next team guess; the same goes for agents. Responsible AI documentation for an agent project should answer five questions:

• Purpose: what may and may not this agent do?
• Data: what data is processed, where is it stored, and how is it cleaned up?
• Limits: what happens when the agent is unsure, and when should it hand off to a human?
• Oversight: who is responsible for the agent's output, and how are they audited?
• Impact: what are the consequences if the agent is wrong, and who bears them?

# Responsible AI - Agent Support
 
1. Agent dilarang menampilkan informasi pribadi pengguna lain.
2. Semua keputusan yang memengaruhi keuangan harus di-escalate ke manusia.
3. Output agent yang dilaporkan pengguna akan direview bulanan.
4. Perubahan prompt utama wajib mendapat persetujuan product owner.
5. Daftar tool dan izinnya direview setiap kali ada rilis baru.

This documentation is not for show — it is the behavior contract between the technical team, the product owner, and the stakeholders. When there is doubt about agent behavior, this document is the reference. Make sure it is always updated whenever a policy changes, rather than being left stale in the repository.

#A Healthy On-Call Culture

Governance and playbooks will not run without a healthy on-call culture. Some practices we have learned from running agents in production: never put incident responsibility on one person without support; decide clearly whether an alert requires a night response or can wait until morning; and run game days regularly — incident simulations — so playbooks are tested before real use. An incident handled calmly is the result of practice, not luck.

#Conclusion

Episode 21 completed the operational side of agents: incident playbooks sequenced from stabilize to learn, rollback and safe mode as safety nets when behavior deviates, governance policies that govern who may deploy and which tools may be used, and responsible AI documentation that serves as a behavior contract for the whole team.

Key takeaways:

• Stabilize the system before debugging — that order is the most often violated.
• Safe mode and rollback must be activatable within seconds.
• Governance without an audit trail is just good intentions.
• Responsible AI documentation is a behavior contract, not decoration.
• Incident practice (game day) makes playbooks tested before real use.

In episode 22 — the closing episode of this series — we will summarize everything into one production package: a final checklist for security, reliability, monitoring, and ethics; day-to-day maintenance for model config, plugins, and user feedback; and how to build an agent architecture ready for the future. See you there!