Async Agents

LLM calls take 2–30 seconds. Blocking synchronous code means one request = one thread = poor resource utilization. Async lets you serve hundreds of concurrent sessions on modest hardware, which is essential for chat APIs and webhook-driven agents.

A synchronous agent is a waiter who stands at one table until the kitchen finishes. An async agent takes new orders while others cook, checking back when each dish is ready.

Scenario

An agent receives 200 concurrent chat requests. Each awaits the LLM without blocking others; parallel tool calls (web search + DB lookup) run via `asyncio.gather`, cutting latency from 8s to 4s.

Solution

In Production Agent Engineering, apply Async Agents to this scenario: An agent receives 200 concurrent chat requests. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Async Agents (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Async Agents happens in the code.

1import asyncio  # import dependencies2from openai import AsyncOpenAI  # import dependencies3
4client = AsyncOpenAI()  # key line for Async Agents5
6async def run_agent(user_msg: str) -> str:  # key line for Async Agents7    # Parallel tool calls8    search, db = await asyncio.gather(  # key line for Async Agents9        web_search(user_msg),10        fetch_user_context(user_msg),11    )12    response = await client.chat.completions.create(  # call the API13        model="gpt-4o-mini",14        messages=[{"role": "user", "content": f"{search}\n{db}\n{user_msg}"}],15    )16    return response.choices[0].message.content  # return the result

• Treating Async Agents as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for async agents

Quick recap — the most important points from this module.