Learn A2A - Observability & Auditing

#Introduction

In episode 14 we mapped threats and designed mitigations. Now a practical question arises: how do we know those mitigations work? The answer: observability. An unmonitored multi-agent network is a black box — you only know a task failed after the user complains.

Observability for A2A has a unique challenge. A single user request can turn into dozens of tasks spread across many agents, in different processes and frameworks. To track it, we need two things: tracing to see a task's journey, and audit logs to record what happened as a permanent record.

This episode's roadmap: we start with why observability matters, then build OpenTelemetry tracing for the task lifecycle, learn parent-child task correlation, and design an audit log that meets compliance needs.

#Why Observability Is Crucial

Imagine a task the orchestrator sends to three agents: one succeeds, one hangs, one returns a strange result. Without telemetry, investigating means opening each service's logs one by one and matching them manually — unrealistic at production scale.

With good tracing, you can answer questions like:

• How long does a task stay in the working state before finishing?
• Which agent is the bottleneck during fan-out?
• Do slow tasks always follow the same routing path?
• At which point does a task fail and why?

A typical multi-agent pattern: a task spawns sub-tasks (parent-child tasks). In episode 17 we'll discuss this fan-out pattern in depth; in this episode we prepare the tools to observe it.

#OpenTelemetry for the Task Lifecycle

OpenTelemetry (OTel) is the observability standard for metrics, logs, and traces. Its core concept is the span: a unit of work with a name, start-end time, and attributes. For A2A, we can map the task lifecycle into a sequence of spans.

Mapping task states to spans:

task.submitted ─▶ task.working ─▶ task.completed
                                  └─▶ task.failed
                └─▶ task.input-required ─▶ task.working ─▶ task.completed

Install the OTel SDK with pip install opentelemetry-api opentelemetry-sdk.

Example of creating a span when the server receives message/send:

instrument.py — spans for the task lifecycle

from opentelemetry import trace
 
tracer = trace.get_tracer("a2a.task")
 
def handle_message_send(payload, parent_ctx=None):
    with tracer.start_as_current_span(
        "task.submitted",
        context=parent_ctx,
        attributes={
            "task.id": payload["params"]["message"]["taskId"],
            "a2a.method": "message/send",
            "peer.agent": payload["headers"].get("agent_id", "unknown"),
        },
    ) as span:
        task_id = payload["params"]["message"]["taskId"]
        result = execute_task(task_id)
        if result.ok:
            span.set_attribute("task.state", "completed")
        else:
            span.set_attribute("task.state", "failed")
            span.record_exception(result.error)
        return result

The attributes on the span become the search key: task.id, a2a.method, and peer.agent let you trace every operation touching one task. Other state transitions — working, input-required — can be represented as child spans or attribute updates as needed.

#Parent-Child Task Correlation

A single user request usually spreads into a chain of tasks. The orchestrator sends a task to agent A, agent A sends a new task to agent B, and so on. To make this chain visible as one flow, each agent must forward the trace context to the next agent.

The standard mechanism is trace context propagation through the HTTP traceparent header. This header carries trace-id, parent-span-id, and a sampling flag. When a client agent sends message/send, it inserts the traceparent header from the current context; the server agent reads that header and makes it the parent context for its spans.

The propagation flow between agents:

user ─▶ orchestrator ──trace-id: 4bf92f57──▶ agent-a ──trace-id: 4bf92f57──▶ agent-b
          │                                    │                              │
        span A                               span B                          span C

Example of sending from a client agent:

curl -X POST https://agent-b.example.com/ \
  -H "Content-Type: application/json" \
  -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
  -d '{"jsonrpc":"2.0","id":"5","method":"message/send","params":{}}'

With correct correlation, tools like Jaeger or Grafana Tempo can show a waterfall displaying the order and duration of each agent in one chain — from submit to complete.

#Audit Log: What to Record

Tracing focuses on debugging and performance. The audit log is the permanent record for accountability and compliance. The difference is crucial: traces can be discarded after a few days, while audit logs often need to be kept for years per regulation.

At minimum, an A2A audit log should record:

• Identity — who sent: the tenant, agent id, or the user who triggered it.
• Action — the method called (message/send, tasks/cancel, and so on).
• Task id and status — all state transitions.
• Decisions — the result the agent chose, including the reason if there is one.
• Data transferred — number of parts, part types, and data size.
• Trace metadata — the trace-id to link the audit log with tracing.

Example of one structured audit log line:

{
  "timestamp": "2026-08-03T09:12:33.102Z",
  "event": "task.completed",
  "task_id": "task-4821",
  "parent_task_id": "task-4800",
  "tenant_id": "acme-corp",
  "actor": "agent:orchestrator-prod",
  "action": "message/send",
  "status": "completed",
  "parts_count": 3,
  "bytes_transferred": 12480,
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Note the parent_task_id: this field links the audit event to its parent task chain, completing the correlation built in tracing.

#Building a Trustworthy Audit Trail

An audit log isn't enough just written to stdout. To be truly trustworthy, it must meet several requirements:

1. Append-only — records must not be edited or deleted after being written.
2. Tamper-evident — changes to records must be detectable, for example with a hash chain between log lines.
3. Trusted time — timestamps must be consistent, usually using NTP or synchronized time sources.

A hash chain works by including the previous line's hash in every new line:

audit.py — log with a hash chain

import hashlib, json
 
prev_hash = ""
 
def write_audit(event: dict):
    global prev_hash
    record = {
        **event,
        "prev_hash": prev_hash,
    }
    payload = json.dumps(record, sort_keys=True)
    current_hash = hashlib.sha256(payload.encode()).hexdigest()
    prev_hash = current_hash
    append_to_audit_store(json.dumps({**record, "hash": current_hash}))

This way, if someone alters a line in the middle of the log, the hashes of all subsequent lines stop matching and the tampering is visible. For the most sensitive data, the audit log can be written to object storage in append mode with restricted access, or even to a system separate from the agent runtime.

#Conclusion

In this episode we built the observability layer for an A2A network. OpenTelemetry maps the task lifecycle from submit to complete into searchable spans, traceparent propagation links parent-child tasks across agents into one chain, and audit logs store a permanent record of who did what with which data — complete with trace metadata to connect the two.

Here's the core takeaway:

• OTel spans map each task state transition and store attributes like task.id and peer.agent for searching.
• The traceparent header must be forwarded between agents so a task chain appears as one trace.
• Audit logs distinguish themselves from tracing: permanent, append-only, and for compliance.
• Every audit record must include identity, action, status, and trace-id.
• Hash chains make the audit log tamper-evident and trustworthy.

With good observability, you know what's happening in the agent network. But monitoring is still easier when the number of agents is small and their addresses are known. What if agents keep growing and their addresses keep changing?

In the next episode, episode 16, we'll discuss Registry & Discovery Service: a service registry as the Agent Card store, an ecosystem agent catalog with dynamic discovery, and routing patterns to choose agents based on capabilities, rating, and latency — like Twilio's latency-aware pattern. See you there!