Top P

Temperature alone can include very low-probability tokens at high values, producing nonsense. Top-p cuts off the long tail of unlikely tokens, keeping outputs coherent while still allowing variety. It's an alternative or complement to temperature.

Top-p is like ordering from a restaurant menu — instead of considering every dish in the city (all tokens), you only pick from the top-rated options that together cover 90% of what people actually order.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Top-p (nucleus sampling) controls output diversity by only sampling from the smallest set of tokens whose cumulative probability exceeds p — dynamically adjusting the candidate pool. Sort tokens by probability descending.
2.Accumulate probabilities until the sum reaches p (e.g., 0.9). Only sample from this 'nucleus' of tokens.
3.At p=1.0, all tokens are candidates. At p=0.1, only the most likely tokens are considered.
4.Unlike temperature (which scales all probabilities), top-p dynamically adjusts the candidate set based on the distribution shape. Typical: top_p=1.0 with temperature=0 for deterministic, or top_p=0.9 with temperature=0.7 for creative but coherent output.

Real Example

Scenario

At top_p=0.9, if the top 3 tokens have probabilities 0.6, 0.25, 0.1 (sum=0.95), only those 3 are candidates. Token 4 at 0.03 is excluded. This prevents the model from picking unlikely words while still allowing variety among plausible options.

What you would do

In Generative AI Foundations, apply Top P to this scenario: At top_p=0. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Top P (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Top P happens in the code.

Top P

1from openai import OpenAI  # import dependencies2
3client = OpenAI()  # create API client4
5response = client.chat.completions.create(  # call the API6    model="gpt-4o-mini",7    messages=[{"role": "user", "content": "Write a creative product description"}],8    temperature=0.7,9    top_p=0.9,  # nucleus sampling10    max_tokens=200,11)12
13# Common production settings:14# Deterministic: temperature=0 (top_p ignored)15# Balanced: temperature=0.7, top_p=0.916# Creative: temperature=1.0, top_p=0.95

Cheat Sheet

Quick recap

quick ref

•top_p=0.9 typical
•Nucleus = cumulative probability cutoff
•temp=0 → top_p ignored
•Balanced: temp 0.7, top_p 0.9
•Adaptive per-token candidate set
•Prevents tail token nonsense

Common Mistakes

✕Setting both temperature and top_p without understanding interaction
✕Using top_p < 0.5 — too restrictive, repetitive output
✕Not setting temperature=0 for deterministic tasks (relying on top_p alone)
✕Changing sampling params without re-evaluating output quality