Realtime Voice

Turn-based STT→LLM→TTS has 3–5s latency and can't handle interruptions. Realtime APIs enable natural conversations with barge-in, emotional tone, and sub-second response times.

Realtime voice is a phone call with a friend — you can interrupt, overlap, and respond instantly. Turn-based voice is exchanging voicemails.

Scenario

Customer calls support line → realtime agent listens, understands issue, queries order DB via function call, speaks empathetic response with order status — all in natural conversation with interruptions.

Solution

In Voice & Multimodal Agents, apply Realtime Voice to this scenario: Customer calls support line → realtime agent listens, understands issue, queries order DB via function call, speaks empathetic response with order status — all in natural conversation with interruptions. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Realtime Voice (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Realtime Voice happens in the code.

1import websockets, json  # import dependencies2
3async def voice_session():  # key line for Realtime Voice4    async with websockets.connect(REALTIME_URL, headers=headers) as ws:  # key line for Realtime Voice5        await ws.send(json.dumps({"type": "session.update", "session": {"modalities": ["audio", "text"]}}))6        async for msg in ws:7            event = json.loads(msg)8            if event["type"] == "response.audio.delta":9                play_audio_chunk(event["delta"])

• Treating Realtime Voice as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for realtime voice

Quick recap — the most important points from this module.