Context Window

Transformer attention is computationally expensive (quadratic with sequence length). Models have hard limits on input size. Exceeding the context window causes errors or truncation. Managing context is a core engineering challenge for chat apps, RAG, and agents.

The context window is like a whiteboard with limited space — you can only fit so much information before you need to erase old content. Smart engineers decide what to keep (recent messages, relevant docs) and what to erase (old conversation turns).

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.The context window is the maximum number of tokens an LLM can process in a single request — including system prompt, conversation history, retrieved documents, and the response.
2.Context window = max input tokens + max output tokens.
3.GPT-4o: 128K tokens.
4.Claude 3: 200K.
5.Everything counts: system prompt, few-shot examples, RAG chunks, conversation history, tool results, and the generated response.

Real Example

Scenario

Your chatbot has a 128K context window. System prompt uses 2K, RAG retrieves 20K of docs, conversation history grows to 100K over 50 turns. You implement sliding window to keep only the last 20K of history, preventing context overflow.

What you would do

In Generative AI Foundations, apply Context Window to this scenario: Your chatbot has a 128K context window. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Context Window (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Context Window happens in the code.

Context Window

1import tiktoken  # import dependencies2
3enc = tiktoken.encoding_for_model("gpt-4o-mini")4MAX_CONTEXT = 128_000  # key line for Context Window5RESERVED_OUTPUT = 4_0966
7def count_tokens(messages: list[dict]) -> int:  # define a reusable function8    return sum(len(enc.encode(m["content"])) for m in messages)  # return the result9
10def trim_history(messages: list[dict], max_tokens: int) -> list[dict]:  # define a reusable function11    system = [m for m in messages if m["role"] == "system"]12    rest = [m for m in messages if m["role"] != "system"]13    while count_tokens(system + rest) > max_tokens and len(rest) > 2:  # key line for Context Window14        rest.pop(0)  # remove oldest non-system message15    return system + rest  # return the result

Cheat Sheet

Quick recap

quick ref

•Count all tokens in request
•Reserve max_tokens for output
•Sliding window for history
•Critical info at start
•Lost in the middle effect
•tiktoken for counting

Common Mistakes

✕Not counting system prompt and RAG tokens in budget
✕Letting conversation history grow unbounded
✕Placing critical instructions in the middle of long context
✕Not reserving enough tokens for the model's response