0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 11 of 21

Decoder

Generation must be autoregressive: predict next token given only previous tokens. The decoder enforces this via causal masking while optionally attending to encoder output for tasks like translation where input and output differ.

Writing a story one sentence at a time — you can only read what you've already written (causal), but you can also refer to your research notes (cross-attention to encoder) for facts.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.The transformer decoder generates output sequences token-by-token using causal self-attention (only past tokens) and cross-attention to encoder representations — powering GPT and machine translation.
  • 2.Decoder layer has three sublayers: (1) Masked Multi-Head Self-Attention — causal mask prevents attending to future tokens, (2) Cross-Attention — Q from decoder, K/V from encoder output (in seq2seq; absent in GPT), (3) FFN.
  • 3.GPT is decoder-only: stacks N decoder layers with only masked self-attention + FFN.
  • 4.During inference, generate one token at a time, append to sequence, repeat.
  • 5.Training uses teacher forcing: ground-truth previous tokens as input.

Real Example

Scenario

GPT generating text: input 'The capital of France is' → decoder processes with causal mask → outputs probability distribution over vocabulary → sample 'Paris' → append → input 'The capital of France is Paris' → continue.

What you would do

In Transformer & ML Foundations, apply Decoder to this scenario: GPT generating text: input 'The capital of France is' → decoder processes with causal mask → outputs probability distribution over vocabulary → sample 'Paris' → append → input 'The capital of France is Paris' → continue. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Decoder (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Decoder happens in the code.

Decoder
1from transformers import GPT2LMHeadModel, GPT2Tokenizer  # import dependencies2
3tokenizer = GPT2Tokenizer.from_pretrained("gpt2")4model = GPT2LMHeadModel.from_pretrained("gpt2")5
6prompt = "The capital of France is"7inputs = tokenizer(prompt, return_tensors="pt")8outputs = model.generate(**inputs, max_new_tokens=10, do_sample=False)9print(tokenizer.decode(outputs[0]))  # show output for debugging

Cheat Sheet

Quick recap

quick ref
  • •Causal mask: tril(ones(n,n))
  • •GPT: decoder-only, no cross-attn
  • •Generate: append token, re-forward
  • •Teacher forcing at train time
  • •LM head: d_model → vocab_size
  • •T5: encoder-decoder architecture

Common Mistakes

  • ✕Removing causal mask during GPT training — information leakage
  • ✕Using teacher forcing at inference time — not possible without ground truth
  • ✕Expecting decoder to see future tokens during generation
  • ✕Confusing GPT (decoder-only) with full encoder-decoder