Observability

• Key points are in the visual diagram above.

Agents are non-deterministic distributed systems. Without observability you're blind to which step failed, why latency spiked, or which prompt version caused regressions.

Agent observability is an airplane cockpit — instruments for altitude (latency), fuel (tokens/cost), and warning lights (errors) so you don't fly blind.

Scenario

Dashboard shows Billing Agent p95 latency doubled after a deploy — trace drill-down reveals new retriever returning 20 chunks instead of 5.

Solution

In Eval Engineering & Observability, apply Observability to this scenario: Dashboard shows Billing Agent p95 latency doubled after a deploy — trace drill-down reveals new retriever returning 20 chunks instead of 5. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Observability (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Observability happens in the code.

1from opentelemetry import trace  # import dependencies2tracer = trace.get_tracer("support-agent")3
4with tracer.start_as_current_span("tool.lookup_invoice") as span:5    span.set_attribute("invoice_id", inv_id)6    result = lookup_invoice(inv_id)

• Treating Observability as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for observability

Quick recap — the most important points from this module.