Streaming

A 500-token response might take 10 seconds to fully generate. Without streaming, users stare at a blank screen for 10 seconds. Streaming shows the first token in ~300ms, keeping users engaged.

Streaming is like watching a live sports broadcast vs waiting for the full match recording. You see the action as it happens, not after it's over.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Streaming delivers LLM tokens as they're generated rather than waiting for the complete response — dramatically improving perceived latency and enabling real-time user experiences. Server-Sent Events (SSE) deliver tokens incrementally.
2.OpenAI: stream=True returns iterator of chunks, each containing a delta (partial token). Client accumulates deltas into full response.
3.Frontend: update UI on each chunk. Async streaming for concurrent requests.
4.LangChain: .stream() and .astream() methods. Considerations: can't parse structured output until complete (stream for display, parse at end), token counting during stream, error handling mid-stream, cancellation via abort signal.

Real Example

Scenario

ChatGPT's typing effect: your prompt hits the API with stream=True. First token appears in ~300ms. Tokens arrive at ~50/sec, each appended to the chat bubble. User reads while generation continues.

What you would do

In LLM Engineering & APIs, apply Streaming to this scenario: ChatGPT's typing effect: your prompt hits the API with stream=True. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Streaming (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Streaming happens in the code.

Streaming

1from openai import OpenAI  # import dependencies2
3client = OpenAI()  # create API client4
5# Sync streaming6stream = client.chat.completions.create(  # call the API7    model="gpt-4o-mini",8    messages=[{"role": "user", "content": "Write a haiku about coding."}],9    stream=True,  # enable token-by-token streaming10)11
12full_response = ""13for chunk in stream:  # iterate over streamed response chunks14    delta = chunk.choices[0].delta.content or ""  # each chunk contains partial text15    print(delta, end="", flush=True)  # show output for debugging16    full_response += delta  # key line for Streaming17
18# Async streaming19import asyncio  # import dependencies20from openai import AsyncOpenAI  # import dependencies21
22async def stream_chat():23    client = AsyncOpenAI()24    stream = await client.chat.completions.create(  # call the API25        model="gpt-4o-mini",26        messages=[{"role": "user", "content": "Count to 5"}],27        stream=True,  # enable token-by-token streaming28    )29    async for chunk in stream:  # iterate over streamed response chunks30        print(chunk.choices[0].delta.content or "", end="")  # show output for debugging

Cheat Sheet

Quick recap

quick ref

•stream=True in API call
•chunk.choices[0].delta.content
•SSE for server-to-client push
•Accumulate for full response
•astream() for async
•Parse structured output after stream ends

Common Mistakes

✕Not flushing output buffer — tokens appear in batches not individually
✕Trying to parse JSON mid-stream — wait for completion
✕Forgetting to handle stream errors/disconnections
✕Using streaming for batch processing — unnecessary overhead