Learn Hermes AI Agent - Resilience & Rate Limiting

#Introduction

In episode 13 your agent knew who was calling it and behaved according to its persona. But there is one reality that cannot be avoided: external services are not always healthy. LLM providers can throttle your requests, tool APIs can go down, networks can be slow. An agent that does not prepare for this will turn from a "smart assistant" into "a pile of errors" when other systems are disrupted.

Here is the roadmap for this episode: facing rate limits and throttling from model providers, installing correct circuit breakers and retry policies, and then building graceful degradation so the agent keeps working (though limited) when other services are down.

#Facing Rate Limits & Throttling

All LLM providers limit how many requests per minute and how many tokens per minute your account may use. When you exceed it, the provider returns status 429 along with a Retry-After header that tells you when you may try again.

The wrong pattern: retrying immediately without delay. That only worsens the throttle. The right pattern: exponential backoff with jitter — a delay that grows longer with each attempt, plus a little randomness so that many agents retrying at once do not form a wave.

export function delayMs(attempt, baseMs = 500) {
  const exponential = baseMs * 2 ** attempt;
  const jitter = Math.random() * exponential * 0.3;
  return exponential + jitter;
}
 
export async function withRetry(fn, { maxAttempts = 5 } = {}) {
  let lastError;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      if (err.status !== 429 && err.status !== 502 && err.status !== 503) {
        throw err;
      }
      await new Promise((r) => setTimeout(r, delayMs(attempt)));
    }
  }
  throw lastError;
}

Notice: only certain errors deserve a retry. Statuses 429, 502, and 503 indicate a temporary problem; status 400 means the request is genuinely wrong and a retry will never succeed.

#Respecting the Retry-After Header

Retry-After tells you the wait time the provider wants. Respecting it is far better than guessing with backoff:

export async function withRetryAfter(fn, maxAttempts = 3) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const res = await fn();
    if (res.status !== 429) return res;
    const seconds = Number(res.headers.get("Retry-After")) || 2;
    await new Promise((r) => setTimeout(r, seconds * 1000));
  }
  throw new Error("rate limit exceeded");
}

For predictable workloads, also consider a token bucket on your side: allocate a per-second request quota to the agent, so you throttle yourself before the provider does. This is more polite to the provider and more stable for agent latency.

#Circuit Breakers

Retry handles momentary disruptions. But if a provider is completely down for five minutes, retrying only wastes time and cost. This is where the circuit breaker comes in: it monitors the failure ratio and "breaks" the request flow after a threshold is reached, then lets the service rest.

The circuit breaker state machine has three states:

• Closed — all requests flow normally.
• Open — after enough failures, requests are rejected outright (fail fast) for a cooldown period.
• Half-open — after the cooldown, a few trial requests are allowed; success closes it again, failure opens it again.

export class CircuitBreaker {
  constructor({ threshold = 5, cooldownMs = 30_000 } = {}) {
    this.threshold = threshold;
    this.cooldownMs = cooldownMs;
    this.failures = 0;
    this.state = "closed";
    this.openedAt = null;
  }
 
  async call(fn) {
    if (this.state === "open") {
      if (Date.now() - this.openedAt > this.cooldownMs) {
        this.state = "half-open";
      } else {
        throw new Error("circuit is open");
      }
    }
    try {
      const result = await fn();
      this.failures = 0;
      this.state = "closed";
      return result;
    } catch (err) {
      this.failures += 1;
      if (this.failures >= this.threshold) {
        this.state = "open";
        this.openedAt = Date.now();
      }
      throw err;
    }
  }
}

Install one circuit breaker per dependency (one for the LLM provider, one for each important tool API), not one for everything — if the weather tool dies, the agent does not need to stop answering non-weather questions. Test its behavior with simulated scenarios via bun run test before touching production.

#Graceful Degradation

When an external service is unavailable and the circuit breaker is open, the agent must not die along with it. It should degrade gracefully:

• Fallback tool — if the weather_api tool fails, try weather_fallback, which has coarser data but still works.
• Partial answer — deliver what can be answered and mention the failed part, instead of rejecting the entire request.
• Cache last-known-good — store the last successful tool result with a TTL; when the source dies, use the old data while flagging that it may be stale.

A common fallback tool strategy in Hermes:

const toolChain = [
  fetchCurrencyLive,
  fetchCurrencyCached,
];
 
export async function getRate(pair) {
  for (const source of toolChain) {
    try {
      return { value: await source(pair), source: source.name };
    } catch {
      continue;
    }
  }
  return { value: null, error: "semua sumber kurs tidak tersedia" };
}

Important point: do not hide degradation. If the answer uses stale cached data, tell the user. Transparency like this preserves trust far better than presenting old data as if it were real-time.

#Measuring All of This

Resilience without measurement is just luck. Track the following metrics for each dependency:

• Error rate — the percentage of requests failing per minute.
• Retry count — how many retries were consumed.
• Circuit state — how long a dependency stayed in the open state.
• Fallback usage — how often a backup tool was used; if it is too frequent, the main service may really be having problems.

These metrics can be fed into agent observability (episode 7) and turned into alarms: a circuit breaker open for more than a few minutes means a service truly needs human attention.

#Conclusion

Episode 14 made your agent shock-resistant. You faced rate limits with jittered exponential backoff and respect for the Retry-After header, absorbed failure bursts with a state-machine circuit breaker, and degraded service gracefully through fallback tools and a last-known-good cache — all while staying transparent about possibly stale data.

Key takeaways:

• Retry only for transient errors — 429, 502, 503 are worth trying; other errors are left to fail.
• Respect Retry-After and add jitter so mass retries do not shake the provider.
• One circuit breaker per dependency, with closed, open, and half-open states.
• Graceful degradation uses fallback tools and a last-known-good cache so the agent does not die along with another service.
• Measure error rate, retry count, and circuit state — resilience without metrics is a guess.

In the next episode 15 we make your agent cheap and fast: Performance Optimization — minimizing prompt costs and token usage, caching model responses and tool results, and profiling runtime latency. See you there!