Phase 1.1 · OptionalOptionalModule 7 of 21

Attention

RNNs process sequences step-by-step, struggling with long-range dependencies and parallelization. Attention lets any position directly access any other position in one step, solving both problems and enabling the parallel training that made LLMs possible.

Reading a contract to answer a question: you don't re-read every word equally. Your eyes jump to relevant clauses (dates, payment terms). Attention is the model learning where to 'look' for each decision it makes.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Attention is a mechanism that lets a model dynamically focus on the most relevant parts of its input when producing each output — the core innovation behind transformers and LLMs. Given query Q, keys K, and values V: Attention(Q,K,V) = softmax(QKᵀ/√d_k)V.
2.Query asks 'what am I looking for?', Key says 'what do I contain?', Value says 'here's my content if selected'. Softmax produces attention weights (sum to 1) over all keys.
3.Weighted sum of values = context-aware output. Scaled dot-product: dividing by √d_k prevents softmax saturation when dimensions are large.
4.Cross-attention: Q from decoder, K/V from encoder (translation). Self-attention: Q, K, V all from same sequence.

Real Example

Scenario

Translating 'The cat sat on the mat' → French. When generating 'chat', the decoder's query attends strongly to encoder keys at 'cat' position. Attention weights might be [0.01, 0.85, 0.02, 0.05, 0.03, 0.04] — focusing on the source word.

What you would do

In Transformer & ML Foundations, apply Attention to this scenario: Translating 'The cat sat on the mat' → French. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Attention (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Attention happens in the code.

Attention

1import torch  # import dependencies2import torch.nn.functional as F  # import dependencies3
4def scaled_dot_product_attention(Q, K, V, mask=None):  # define a reusable function5    d_k = Q.size(-1)6    scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)7    if mask is not None:8        scores = scores.masked_fill(mask == 0, float('-inf'))9    weights = F.softmax(scores, dim=-1)  # key line for Attention10    return torch.matmul(weights, V), weights  # return the result11
12# Example: 1 head, seq_len=4, d_k=813Q = K = V = torch.randn(1, 4, 8)14output, weights = scaled_dot_product_attention(Q, K, V)  # key line for Attention15print(f"Output: {output.shape}, Weights: {weights.shape}")  # show output for debugging

Cheat Sheet

Quick recap

quick ref

•Attn = softmax(QKᵀ/√d_k)V
•Q=query, K=key, V=value
•Self-attn: same sequence
•Cross-attn: different sequences
•Mask: -inf for blocked positions
•O(n²d) complexity

Common Mistakes

✕Forgetting to scale by sqrt(d_k) — unstable softmax
✕Not applying causal mask in decoder self-attention
✕Confusing attention weights with model confidence
✕Assuming attention always aligns with human intuition