Learn Hermes AI Agent - CI/CD & Release for Agents

#Introduction

In episode 18 you made an adaptive and reflective agent — one that can change its own strategy based on feedback. But remember our closing message: any change must be tested in a small group and must be rollback-able. The question now is, how do you test that change automatically, repeatedly, and safely before it reaches real users?

The answer is a CI/CD and release pipeline designed specifically for agents. Agents are unique compared to ordinary applications: their output is non-deterministic, their dependencies include model configuration and prompts, and "deploying" often means changing behavior, not just swapping code. Episode 19 covers three foundations of that pipeline: behavior testing in CI, safe deployment, and traceable versioning. Here is the roadmap for this episode:

• Why an agent pipeline differs from an ordinary application pipeline.
• Testing agent behavior in CI with automated evaluation.
• Deploying code and model configuration safely.
• Versioning agent profiles and tools.

#Why an Agent Pipeline Is Different

A conventional pipeline tests with deterministic expectations: the same input must produce the same output. Agents are not like that — the same model can answer differently to the same question, and tool output changes all the time. This is not a bug; it is their nature. So the testing strategy must shift:

• From asserting exact output to asserting invariants: the agent must not use forbidden tools, must not leak personal data, must complete the task within step limits.
• Add dataset-based evaluation: a collection of test scenarios graded with scores, not binary right/wrong.
• Use deterministic seeds and fixed models for smoke tests, and varied models for tests that judge quality.

bun install --frozen-lockfile
bun run lint
bun run eval --suite regressions
bun run eval --suite safety --model gpt-4o-mini

The safety test here is not picking the model because it is cheap, but because a smaller model fails more often on edge cases — so it is a more sensitive problem detector. If the safety suite passes on the small model, it will very likely also pass on the big model.

#Building an Eval Harness in CI

An eval harness is the code that runs the agent against test scenarios and produces a score. It must be runnable both locally and in CI so the results are consistent. Here is a simple harness shape using the suite you already saw in episode 18:

import { HermesAgent, evaluateSuite } from "@hermes/sdk";
 
const agent = new HermesAgent({ profile: process.env.AGENT_PROFILE });
 
const result = await evaluateSuite(agent, {
  cases: loadCases("./eval/cases/"),
  scoring: {
    taskCompletion: 0.5,
    toolDiscipline: 0.3,
    latency: 0.2,
  },
});
 
if (result.totalScore < 0.85) {
  process.exit(1);
}

Run this harness in CI as a gate: if the score is below the threshold, the pipeline stops and there is no deploy. The minimum threshold should be raised gradually as the test dataset grows — so "passing CI" does not become an overgrown shrub with ever-declining quality.

#Deploying Code and Model Configuration Safely

Once CI passes, it is time to deploy. The agent's peculiarity: what changes is not only code, but also the model configuration — which model is used, temperature, top-p, the system prompt, the tool list. This configuration must be deployed as a versioned artifact, not changed manually on the server.

version: 1.4.0
model:
  id: gpt-4o
  temperature: 0.3
  max_tokens: 2048
profile: ./profiles/support.ts
tools:
  - web-search
  - db-readonly
  - escalate-human
memory:
  ttl_days: 30

Deploying model configuration carries the same risk as deploying code: a new version can lower quality. That is why you use the blue-green or canary pattern, rather than switching off the old version at once. Hermes deployment orchestration already supports both paths at the same time:

hermes deploy --env production --version 1.4.0 --canary 10%
hermes promote --env production --version 1.4.0

The first command sends the new version to 10 percent of traffic; the second only promotes it fully once it is confirmed healthy. If a problem appears along the way, hermes rollback --env production returns traffic to version 1.3.9 within seconds.

#Versioning Agent Profiles and Tools

To be rollback-able, every part that affects behavior must have a version. In Hermes, there are three artifacts that must be versioned: the profile (persona and system prompt), the tool set (tool list and permissions), and the model config. The three are tied into one release tag, so restoring a version means restoring the behavior completely.

hermes tag --profile support@1.4.0 --tools @hermes/tools@2.1.3 --config 1.4.0

For tracing history, use ordinary semver — agent releases follow the feat:, fix:, and breaking: pattern. Because agents are non-deterministic, add one special convention: every release stores its evaluation record (dataset, scores, models) as an attachment. That way, when a regression happens, you can answer two key questions: what changed, and how bad the change is.

#Conclusion

Episode 19 turned the agent release process from "deploy code and pray" into a provable pipeline: behavior is tested in CI with an eval harness that produces scores, deployment uses a canary pattern that protects traffic, and every part — profile, tools, model config — is versioned and bound into a release that can be rolled back.

Key takeaways:

• Test agents with invariants and scores, not with exact output expectations.
• An eval harness must be a gate that stops the pipeline when the score is below the threshold.
• Model configuration is a deploy artifact, not a manual setting.
• Deploy with canary and roll back within seconds.
• Version every artifact that affects behavior, and bind them all into one release.

In the next episode 20 we raise observability to production scale: success metrics, dashboards for agent activity and tool usage, and alerts that wake you up when actions fail or behavior changes suspiciously. See you there!