Learn A2A - Performance & Scale

#Introduction

In episode 17 you assembled advanced task patterns: fan-out and fan-in between agents, human-in-the-loop flows through the input-required state, up to idempotency, retry, and checkpoints for long-lived tasks. All those patterns work well when the task count is still in the dozens. The problem appears when tasks reach thousands per second: patterns that are logically correct can fall apart technically.

Episode 18 is the next stepping stone: keeping latency and stability as scale grows. We'll discuss connection pooling so TCP connections aren't opened for every task, agent card caching so discovery traffic doesn't weigh you down, load balancing for many remote agent replicas, the streaming vs polling tradeoff, payload minimization, and when to switch to the gRPC binding for high throughput.

#Connection Pooling: Don't Open a New Connection Every Task

Every TCP and TLS handshake costs tens of milliseconds — cheap once, expensive a thousand times. A naively written A2A client usually opens a new connection per task. The solution is connection pooling: one HTTP keep-alive connection reused for many requests.

pool-httpx.py

import httpx
from a2a_sdk import A2AClient
 
transport = httpx.AsyncHTTPTransport(
    limits=httpx.Limits(max_connections=50, max_keepalive_connections=20)
)
 
async with httpx.AsyncClient(transport=transport, timeout=30.0) as client:
    a2a = A2AClient(client)
    for task in daftar_task:
        result = await a2a.send_task(task)

Note the two httpx.Limits parameters: max_connections limits the total connections that may be open, while max_keepalive_connections determines how many idle connections are kept alive for later use. These values must match the connection limits on the remote agent's side — if the client opens 50 connections but the server only holds 20, the rest queue in the kernel. Rule of thumb: start with 10-20 keepalives per host, then measure.

#Caching the Agent Card: Reduce Discovery Traffic

Before a task is sent, the client must fetch the agent card to know the remote agent's capabilities. If it's fetched for every task, that wastes bandwidth and adds latency to each request. Agent cards rarely change — even one from a week ago is still valid. The solution: cache with a TTL.

cache-agent-card.py

import time
 
class AgentCardCache:
    def __init__(self, ttl_seconds=3600):
        self.ttl_seconds = ttl_seconds
        self._store = {}
 
    def get(self, url: str):
        entry = self._store.get(url)
        if entry and time.time() - entry["ts"] < self.ttl_seconds:
            return entry["card"]
        return None
 
    def set(self, url: str, card: dict):
        self._store[url] = {"card": card, "ts": time.time()}

Once the card is cached, the client can send tasks/send directly without waiting for a discovery round-trip. The TTL should be shorter than your agent's release schedule: if agents are deployed several times a day, lower the TTL to 300 seconds; if they rarely change, 3600 seconds is safe. For signed agent cards (already discussed in episode 9 and deepened in episode 21), the cache should also store the signature verification result so the cryptographic verification isn't repeated.

#Load Balancing Many Remote Agents

When a single remote agent can no longer handle the load, the answer is replicas — and replicas need a load balancer. A2A's specific challenge: tasks are stateful. A task already running on replica A must continue on replica A; if it's routed to replica B, the client won't find its state.

upstream a2a-agent {
    least_conn;
    server agent-a.internal:3000;
    server agent-b.internal:3000;
    server agent-c.internal:3000;
}
 
server {
    listen 443 ssl;
    location / {
        proxy_pass http://a2a-agent;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Cookie $http_cookie;
    }
}

The least_conn strategy fits because A2A tasks have varying durations — a replica that receives heavy tasks will be quieter on subsequent connections. But the real key is sticky sessions: if task state is stored per-replica, make sure the same client always lands on the same replica. A more robust alternative is storing task state in a shared store (Redis, database) so any replica can continue — then the load balancer can use a pure strategy without stickiness. And don't forget health checks using tasks/get with a dummy task, not just a TCP ping.

#Streaming vs Polling: The Tradeoff

Episode 7 covered SSE streaming and push notifications. At the performance level, this decision becomes a real tradeoff:

• Streaming (SSE): lowest per-event latency — the client sees result deltas as soon as they're available. But long connections are maintained, so they hold connection pool slots longer and are prone to dropping at proxy idle timeouts.
• Polling: one connection per request, short and done. Simple, passes any proxy, and natural for retries. The cost: average latency rises by half the polling interval.

polling-backoff.py

async def poll_task(client, task_id: str, timeout: int = 120):
    deadline = time.time() + timeout
    delay = 1.0
    while time.time() < deadline:
        task = await client.get_task(task_id)
        if task.status not in {"submitted", "working"}:
            return task
        await asyncio.sleep(delay)
        delay = min(delay * 1.5, 10.0)
    raise TimeoutError("task terlalu lama")

Polling with exponential backoff is the best compromise for the majority of cases: short tasks are detected quickly, long tasks don't guess a fixed interval. Use streaming only when real-time interactivity is worth it — for example, an agent showing partial results to a waiting user.

#Payload Minimization

Large payloads are the silent enemy of latency. Network queues fill with unnecessary data, and JSON parsing eats CPU on both sides. Three techniques with immediate impact:

• File parts with URLs, not inline dataURIs — file parts have two forms: dataURI (the byte content is embedded) or URL (a reference). For large files, send the URL; the file content is downloaded separately, in parallel, and can be cached.
• Compact structured parts — use the smallest JSON schema that suffices, rather than copying an entire database record. Send the id, name, and price only, not 40 columns.
• Avoid context duplication — if a message repeats, send it once and reference it by id, don't repeat its content in every message.

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "message/send",
  "params": {
    "message": {
      "role": "user",
      "parts": [
        { "type": "text", "text": "Ambil detail order ini" },
        {
          "type": "file",
          "url": "https://cdn.internal/orders/8321.json",
          "mimeType": "application/json"
        }
      ]
    }
  }
}

Note the file part above: a 2 MB file is not sent over the A2A wire — only its URL. This is what separates systems capable of hundreds of tasks per second from ones that stall at 20 tasks per second.

#gRPC for High Throughput

If pooling, small payloads, and polling still aren't enough — for example, a swarm of agents sending tasks all day at very high volume — it's time for the gRPC binding. Its main advantages:

• Multiplexing: many streams in one HTTP/2 connection, without head-of-line blocking.
• Binary protobuf: payloads far smaller than JSON and much cheaper to parse.
• Native backpressure: the data flow stops automatically when the receiver is slow, so memory doesn't blow up.

syntax = "proto3";
 
message Task {
  string id = 1;
  string status = 2;
  repeated string messages = 3;
}
 
message TaskQuery {
  string id = 1;
}
 
service A2AAgentService {
  rpc SendMessage(stream TaskQuery) returns (stream Task);
  rpc GetTask(TaskQuery) returns (Task);
}

gRPC isn't a replacement — it reuses the same server handlers as the HTTP/JSON binding (we discussed the gateway in episode 11). A healthy strategy: make HTTP/JSON the public default, and gRPC the internal path between agents that know each other. Busy internal clients can talk gRPC directly, while the external ecosystem keeps using standard JSON-RPC.

#Conclusion

Episode 18 raises the level from "tasks run correctly" to "tasks run correctly in the thousands". Connection pooling cuts handshake costs, agent card caching eliminates repeated discovery traffic, load balancing spreads the load while preserving task state, backoff polling balances latency and simplicity, compact payloads free up bandwidth, and gRPC opens a high-throughput path between internal agents.

Here's the core takeaway:

• Connection pooling and agent card caching are the first two cheapest performance wins.
• A2A tasks are stateful — design load balancing and task state storage together.
• Streaming only for cases that need per-event latency; backoff polling suffices for the rest.
• Minimize the wire: URLs for files, compact structured parts, and no context duplication.
• gRPC for busy internal paths; HTTP/JSON remains the public language.

In episode 19 we enter more interesting territory: the x402 & commerce ecosystem — how agents can pay other agents through the HTTP 402 extension, mandates, and approvals. See you there!