Latency

Users abandon slow agents. Interviewers ask how you'd hit p95 < 3s for chat and < 30s for complex tasks.

Latency is wait time at a restaurant — even great food loses customers if every course takes an hour.

Scenario

RAG bot p95 was 8s. Parallel retrieval + gpt-4o-mini for simple queries + streaming cuts p95 to 2.1s.

Solution

In Production Agent Engineering, apply Latency to this scenario: RAG bot p95 was 8s. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Latency (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Latency happens in the code.

1# Latency — minimal example2from openai import OpenAI3
4client = OpenAI()  # create API client5
6# Ask the model to explain this topic7response = client.chat.completions.create(  # core API call for Latency8    model="gpt-4o-mini",9    messages=[10        {"role": "system", "content": "You explain latency clearly."},11        {"role": "user", "content": f"What is latency?"},12    ],13    temperature=0,14)15print(response.choices[0].message.content)  # show output for debugging

• Treating Latency as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for latency

Quick recap — the most important points from this module.