Learn Hermes AI Agent - Performance Optimization

#Introduction

In episode 14 you prepared the agent to survive an imperfect network: exponential backoff when rate-limited, circuit breakers to isolate dead dependencies, and graceful degradation so other services collapsing does not take the agent down too. Your agent is now resilient. But resilient is not enough — an agent that burns twice as many tokens and responds twice as slowly also eats money.

Here is the roadmap for this episode: cutting prompt costs and token usage, leveraging caching for model responses and tool results, and then profiling runtime latency to find the real bottlenecks.

#Calculating the Cost per Conversation

Every turn pays for two things: input tokens (prompt, history, tool results) and output tokens (response). Cost is calculated per token per model, and every character sent to the model is counted. So the only way to save is to reduce the number of tokens passing through — not to negotiate the model price.

Start with an explicit token budget:

budget:
  max_input_tokens: 12000
  max_output_tokens: 800
  summarize_history_at: 9000
  hard_stop_on_exceed: true

max_input_tokens limits the total prompt that may enter the model; when it is exceeded, the request is constrained. max_output_tokens forces concise answers and at the same time lowers latency. summarize_history_at triggers a history summary (remember episode 8) before the prompt balloons. If in episode 9 you managed the prompt window manually, here Hermes manages it automatically based on those thresholds.

#Cutting Tokens in the Prompt and Context

The model reads everything you give it. If the system prompt contains 40 lines of instructions that never affect the output, that is a fixed cost on every turn. The principle: instructions that never change the output are wasted tokens.

Routine audits you can do:

• Drop instruction lines that were not used in the past week — you can check this from the event logs in episode 7.
• Move details to retrieval, rather than always cramming the entire knowledge base into the prompt.
• Summarize older conversation history and only include the turns relevant to the current question.
• Limit answer length; uncontrolled output is a double waste, in cost and in latency.

Do not forget model routing: simple tasks do not need a big model. Hermes allows routing rules based on task type:

routing:
  default: gpt-4o
  rules:
    - match: type == "translation" or type == "summarize"
      model: gpt-4o-mini
    - match: type == "code_analysis"
      model: gpt-4o

With this routing, most lightweight requests use a small model, and the big model is only called when it is truly needed. This is the biggest saving with the smallest change.

#Caching Model Responses

The next most common waste is duplicated work: hundreds of users ask nearly the same thing, and every time the model answers from scratch. The solution is a semantic cache — store answers, then on the next request match them by meaning similarity, not exact string matching.

import { embed, vectorSearch } from "@hermes/cache";
 
const THRESHOLD = 0.92;
const TTL_MS = 24 * 60 * 60 * 1000;
 
export async function cachedReply(agent, input) {
  const query = await embed(input);
  const hit = await vectorSearch("reply_cache", query, {
    threshold: THRESHOLD,
    ttl: TTL_MS,
  });
  if (hit) return { reply: hit.reply, fromCache: true };
 
  const reply = await agent.run(input);
  await vectorSearch("reply_cache", query, {
    store: { reply },
    ttl: TTL_MS,
  });
  return { reply, fromCache: false };
}

The query is embedded into a vector and then compared against stored answers. Similarity above the 0.92 threshold is considered the same question, and the old answer is returned without a model call. Cache responses are also much faster — this cuts latency and cost at the same time.

#Caching Tool Output

Tool calls are often the slowest part and the most frequently rate-limited, as you handled in episode 14. Deterministic tools — fetching exchange rates, checking weather, reading config — may be cached with a TTL matching their data freshness.

const ttlCache = new Map();
 
export async function cachedTool(name, args, { ttlMs = 60_000 } = {}) {
  const key = name + ":" + JSON.stringify(args);
  const entry = ttlCache.get(key);
  if (entry && Date.now() - entry.at < ttlMs) return entry.value;
 
  const value = await invoke(name, args);
  ttlCache.set(key, { value, at: Date.now() });
  return value;
}

The golden rule: only cache read-only tools. get_weather, fetch_currency, and query_catalog are fine; send_email, create_order, and delete_record never. And as in episode 14, mark responses that come from the cache so both the agent and the user know the data may be stale.

#Profiling Runtime Latency

Once costs are down, it is speed's turn. Before optimizing, you must know where the time is actually spent. Agent latency can be split into three phases: plan (assembling steps), tool (calling tools), and respond (generating the answer).

export async function profileRun(agent, task) {
  const t0 = performance.now();
  const plan = await agent.plan(task);
  const t1 = performance.now();
  const result = await agent.execute(plan);
  const t2 = performance.now();
  const reply = await agent.respond(result);
  const t3 = performance.now();
 
  return {
    reply,
    phases: {
      planMs: Math.round(t1 - t0),
      toolMs: Math.round(t2 - t1),
      respondMs: Math.round(t3 - t2),
    },
  };
}

Read the results with a cool head:

• planMs dominates: the model is thinking too much — reduce max_output_tokens, use a faster model, or simplify the instructions.
• toolMs dominates: optimize the tool itself or strengthen the tool cache above.
• respondMs dominates: reduce answer length or route to a small model.

For an overall picture across sessions, Hermes provides a built-in aggregator: run hermes perf report after the agent has served several sessions. The report contains p50 and p95 for each phase. A large gap between p50 and p95 indicates some sessions are stuck — often because of one slow tool or a cache miss.

#Conclusion

Episode 15 made your agent cheap and fast. You set a token budget and slimmed the prompt, stored duplicated results through a semantic cache and TTL tool caches, and profiled latency per phase to know exactly where optimization should be aimed. The combination of a disciplined token budget and the right caching can cut costs by tens of percent without changing answer quality.

Key takeaways:

• Tokens are money — calculate the cost per turn and set max_input_tokens and max_output_tokens.
• Streamline the prompt; unused instructions are a fixed cost on every turn.
• Route light tasks to small models, and keep big models for heavy tasks.
• Semantic cache for repeated questions, tool cache for deterministic read-only tools.
• Per-phase profiling answers "where is the slowness" before you optimize by guessing.

In the next episode 16 we empower the agent with capabilities that are not built in: Agent Customization & Extensions — writing custom actions and plugins, extending capabilities with domain-specific tools, and packaging agent modules so they can be reused across projects. See you there!