0
0What is AI1ML vs DL vs GenAI2LLMs3Transformers4Tokens5Tokenization6Embeddings7Similarity Search8Prompt Engineering9Temperature10Top P11Context Window12Hallucination13Model Providers14Open vs Closed Models
Generative AI Foundations
Phase 1Module 6 of 15

Tokenization

Models need fixed-size integer inputs. Tokenization bridges human-readable text and model-compatible sequences. Different models use different tokenizers, so the same text produces different token counts across providers.

Tokenization is like a translator between human language and machine language — it breaks sentences into a vocabulary the model understands, and reassembles the model's output back into readable text.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Tokenization is the process of converting raw text into token IDs that LLMs can process — and detokenizing outputs back to readable text.
  • 2.Common algorithms: BPE (Byte Pair Encoding) merges frequent character pairs iteratively, WordPiece (used by BERT) splits on subwords with ## prefix, SentencePiece (used by Llama) treats input as raw bytes.
  • 3.Each model has a fixed vocabulary (32K-100K tokens).
  • 4.Special tokens mark boundaries: <|endoftext|>, [CLS], [SEP].
  • 5.The tokenizer is tied to the model — using the wrong tokenizer corrupts input.

Real Example

Scenario

The word 'tokenization' might tokenize as ['token', 'ization'] in GPT models but ['tokeni', '##zation'] in BERT. Same word, different splits — which is why you must use the model's own tokenizer.

What you would do

In Generative AI Foundations, apply Tokenization to this scenario: The word 'tokenization' might tokenize as ['token', 'ization'] in GPT models but ['tokeni', '##zation'] in BERT. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Tokenization (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Tokenization happens in the code.

Tokenization
1from transformers import AutoTokenizer  # import dependencies2
3tokenizer = AutoTokenizer.from_pretrained("gpt2")  # key line for Tokenization4
5text = "Tokenization splits text into subword units"6encoded = tokenizer(text)  # key line for Tokenization7print(f"Token IDs: {encoded['input_ids']}")  # show output for debugging8print(f"Tokens: {tokenizer.convert_ids_to_tokens(encoded['input_ids'])}")  # show output for debugging9print(f"Decoded: {tokenizer.decode(encoded['input_ids'])}")  # show output for debugging10
11# Compare tokenization across models12for model in ["gpt2", "bert-base-uncased"]:13    tok = AutoTokenizer.from_pretrained(model)  # key line for Tokenization14    ids = tok.encode(text)  # key line for Tokenization15    print(f"{model}: {len(ids)} tokens")  # show output for debugging

Cheat Sheet

Quick recap

quick ref
  • •AutoTokenizer.from_pretrained()
  • •encode() → token IDs
  • •decode() → text
  • •BPE = merge frequent pairs
  • •Special tokens for structure
  • •Match tokenizer to model

Common Mistakes

  • ✕Using the wrong tokenizer for a model
  • ✕Not handling unicode and emoji in tokenization
  • ✕Assuming tokenization is reversible without artifacts
  • ✕Ignoring special tokens when counting context usage