Tokens

Neural networks operate on numbers, not text. Tokenization converts text to integer sequences the model can process. Token counts determine API costs, context window usage, and response speed — making them a critical engineering concern.

Tokens are like individual LEGO bricks — the word 'understanding' might be one brick, but 'uncharacteristically' might be four smaller bricks snapped together. The model builds meaning brick by brick.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Tokens are the atomic units LLMs process — pieces of text (words, subwords, or characters) that everything is measured, billed, and limited by. LLMs don't see words — they see token IDs.
2.Common tokenizers (BPE, SentencePiece) split text into subword units balancing vocabulary size and sequence length. English averages ~4 characters per token.
3.Costs are per-token (input + output). Context windows are token-limited (e.g., 128K tokens).
4.Longer prompts = more cost + slower inference. Token count affects what fits in context: a 100-page PDF might be 50K+ tokens.

Real Example

Scenario

The prompt 'Explain quantum computing in simple terms' is ~7 tokens. The 500-word response is ~650 tokens. At $0.15/1M input tokens, the prompt costs fractions of a cent — but at 10K requests/day, token economics matter.

What you would do

In Generative AI Foundations, apply Tokens to this scenario: The prompt 'Explain quantum computing in simple terms' is ~7 tokens. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Tokens (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Tokens happens in the code.

Tokens

1import tiktoken  # import dependencies2
3# Load the tokenizer for your model4enc = tiktoken.encoding_for_model("gpt-4o-mini")5
6text = "Explain quantum computing in simple terms"7
8# Text → token IDs (this is where tokenization happens)9tokens = enc.encode(text)  # text → token IDs — this is where tokenization happens10print(f"Token count: {len(tokens)}")  # show output for debugging11print(f"Token IDs: {tokens}")  # show output for debugging12print(f"Decoded: {enc.decode(tokens)}")  # convert IDs back to text13
14# Estimate API cost from token counts15input_tokens = len(enc.encode(system_prompt + user_message))16output_tokens = 500  # max_tokens setting17cost = (input_tokens * 0.15 + output_tokens * 0.60) / 1_000_00018print(f"Estimated cost: ${cost:.6f}")  # show output for debugging

Cheat Sheet

Quick recap

quick ref

•tiktoken.encoding_for_model()
•len(enc.encode(text))
•~4 chars/token English
•input + output = total cost
•Code ≈ 2-3x prose tokens
•max_tokens caps output

Common Mistakes

✕Not counting tokens before sending large documents to LLMs
✕Assuming 1 word = 1 token
✕Ignoring system prompt and history in token budget
✕Setting max_tokens too high — paying for unused capacity