Learn Hermes AI Agent - Observability at Scale

#Introduction

Episode 19 taught you how to release agents with a safe pipeline: testable and rollback-able. But a safe pipeline is useless if no one knows whether the released agent is actually working well in the field. The logging in episode 7 is still enough for one agent; for many agents serving thousands of conversations, you need full-scale observability.

Episode 20 covers the three pillars of production observability: metrics that answer "is the agent succeeding", dashboards that display activity and tool usage in real time, and alerts that flag failures and anomalous behavior before they become major incidents. Here is the roadmap for this episode:

• Defining success metrics: task completion, error rate, latency.
• Building dashboards for agent activity and tool usage.
• Alerts for failed actions and anomalous behavior.
• Connecting it all with tracing and correlation.

#The Real Success Metrics

Most applications are measured with uptime and HTTP status codes. Agents cannot be measured that way — a request can be technically successful while answering with wrong information. That is why the three core metrics for agents must be understood differently from ordinary applications:

Task completion is the metric that most distinguishes agents from other applications. To measure it, the agent needs to tag its own conversations with the final outcome status, then store it together with the telemetry:

import { metrics } from "@hermes/telemetry";
 
await agent.run(input, {
  onComplete(result) {
    metrics.histogram("agent.latency_ms", result.latencyMs);
    metrics.increment("agent.tasks_total", { outcome: result.outcome });
    metrics.observeConfidence(result.confidence);
  },
});

Notice the metric pattern above: always label the metric (outcome, confidence) so the dashboard can be broken down per category. An unlabeled metric only tells you "something is wrong"; a labeled metric tells you "where it is wrong".

#Dashboards for Agent Activity and Tool Usage

Raw metrics are useless until visualized. A good agent dashboard has two layers: a summary layer for the operations team, and a detail layer for debugging. For dashboards, Hermes exports metrics in the Prometheus format, which can be consumed directly by Grafana or your favorite observability stack.

scrape_configs:
  - job_name: hermes-agents
    metrics_path: /metrics
    static_configs:
      - targets: ["agent-api:9100"]

In Grafana, arrange panels that answer business questions, not just display charts:

• Task completion rate per profile — which agent fails most often.
• Error rate per tool — which tool causes problems most often.
• Latency breakdown per phase — is the slowness in the model, the tool, or reflection.
• Top tool usage — which tools are called most often, and whether any tool is almost never used (possibly a wrong configuration).
• Confidence distribution — whether the agent is increasingly answering with low confidence.

One dashboard for everything is a recipe for failure. Make at most three dashboards: one for executives (summary), one for on-call (active alerts), and one for engineering (trace details).

#Alerts for Failed Actions and Anomalous Behavior

Dashboards are only useful if someone is watching them. At night, no one watches — so you need alerts that work tirelessly. There are two classes of alerts for agents: reactive alerts for failures, and proactive alerts for anomalies that have not yet become failures.

groups:
  - name: agent-health
    rules:
      - alert: HighErrorRate
        expr: rate(agent_errors_total[5m]) > 0.2
        for: 10m
        labels:
          severity: page
      - alert: ToolLatencySpike
        expr: histogram_quantile(0.95, rate(agent_tool_latency_seconds_bucket[5m])) > 5
        for: 5m
        labels:
          severity: warning
      - alert: ConfidenceDrop
        expr: avg(agent_confidence{phase="final"}) < 0.5
        for: 30m
        labels:
          severity: warning

The rules above translate the three signals we have already discussed: high error rate, ballooning tool latency, and plummeting confidence. For proactive alerts, also install simple anomaly detection: compare current metrics with last week's baseline, and alert if they deviate by several standard deviations — for example, the number of escalations suddenly doubling, or the tool usage pattern changing drastically with no release at all.

#Tracing and Cross-Service Correlation

Metrics tell you that there is a problem, dashboards tell you what is wrong, but to find why, you need tracing. Every agent conversation must have a trace ID that follows its entire journey: from input, planning, each tool call, reflection, to the final output.

hermes log stream --trace-id conv_8f3a --format json

This trace ID must also appear in the logs of other applications the agent calls — otherwise it is impossible to connect a user complaint with the wrong tool step. With tracing, incident investigation changes from "watch the dashboard" to "follow the conversation trail": which step produced the wrong answer, which tool returned weird data, and in which phase the agent decided something wrong. This is exactly the provision you need for the next episode on Ops & Governance.

#Conclusion

Episode 20 built the foundation of production-scale observability: success metrics that distinguish technical success from functional success, layered dashboards that answer different questions at every level, reactive and proactive alerts that guard the night, and tracing that connects everything into one complete story per conversation.

Key takeaways:

• Task completion is the agent's most characteristic metric — measure the final result, not just the request status.
• Label the metrics so problems can be traced back to a category.
• At most three dashboards: executive, on-call, and engineering.
• Alerts must be balanced: too sensitive makes the team blind, too loose makes the system vulnerable.
• The trace ID must run through the whole conversation and every service called.

In the next episode 21 we use all these observability signals to run daily operations: incident playbooks, rollback and safe mode, governance policies for agent usage, and documentation of responsible AI practices. See you there!