0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 13 of 21

Transformer Architecture

RNNs/LSTMs were sequential (slow to train), struggled with long-range dependencies, and hard to parallelize on GPUs. Transformers solve all three: parallelizable, direct long-range connections via attention, and infinitely stackable.

RNNs are like reading a book one word at a time, remembering in your head. Transformers are like having the entire book spread on a table, with highlighters connecting related passages instantly — all at once.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.The transformer is the foundational architecture behind all modern LLMs — replacing recurrence with stacked attention and feed-forward layers that process sequences in parallel. Original 'Attention Is All You Need' (2017): Encoder-Decoder for translation.
  • 2.Modern LLM variants: Encoder-only (BERT), Decoder-only (GPT), Encoder-Decoder (T5). Core building block: Transformer Layer = Attention + FFN + Residual + LayerNorm.
  • 3.Decoder-only (GPT): N × (Masked MHA → Add&Norm → FFN → Add&Norm) → LM Head. Key hyperparameters: d_model (hidden dim), num_layers, num_heads, d_ff (FFN inner dim, typically 4×d_model), vocab_size, max_seq_len.
  • 4.Scaling laws: more parameters + more data = better performance predictably.

Real Example

Scenario

GPT-3: 96 layers, d_model=12288, 96 heads, d_ff=49152, vocab=50257. Input 'Write a poem about' → 96 transformer blocks process → LM head outputs logits over 50257 tokens → 'the' sampled → append and repeat.

What you would do

In Transformer & ML Foundations, apply Transformer Architecture to this scenario: GPT-3: 96 layers, d_model=12288, 96 heads, d_ff=49152, vocab=50257. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Transformer Architecture (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Transformer Architecture happens in the code.

Transformer Architecture
1from transformers import AutoModelForCausalLM, AutoTokenizer  # import dependencies2
3model_name = "gpt2"  # 12 layers, 768 dim, 12 heads4model = AutoModelForCausalLM.from_pretrained(model_name)5tokenizer = AutoTokenizer.from_pretrained(model_name)6
7# Model architecture summary8print(f"Layers: {model.config.n_layer}")  # show output for debugging9print(f"Hidden size: {model.config.n_embd}")  # show output for debugging10print(f"Attention heads: {model.config.n_head}")  # show output for debugging11print(f"FFN dim: {model.config.n_inner or model.config.n_embd * 4}")  # show output for debugging12print(f"Vocab size: {model.config.vocab_size}")  # show output for debugging

Cheat Sheet

Quick recap

quick ref
  • •Block: Attn → Norm → FFN → Norm
  • •GPT: decoder-only, causal mask
  • •BERT: encoder-only, bidirectional
  • •d_ff = 4 × d_model (typical)
  • •Residual: x + sublayer(x)
  • •Scaling laws: params ∝ performance

Common Mistakes

  • ✕Confusing encoder-only with decoder-only use cases
  • ✕Omitting residual connections in custom implementations
  • ✕Not accounting for FFN's 4× memory expansion
  • ✕Assuming all transformers have encoder AND decoder