Phase 1.1 · OptionalOptionalModule 8 of 21

Self Attention

Language understanding requires knowing how words relate to each other ('bank' near 'river' vs 'money'). Self-attention computes these relationships for all pairs simultaneously, replacing sequential RNN processing with parallel computation.

A group discussion where everyone can hear everyone else at once. Each person (token) listens to the whole conversation and decides which speakers are most relevant to their own contribution.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Self-attention lets every token in a sequence attend to every other token in the same sequence — enabling each position to gather context from the entire input in parallel.
2.Input X (seq_len × d_model) is projected to Q = XW_Q, K = XW_K, V = XW_V via learned weight matrices.
3.Each row of Q is one token's query; each row of K is one token's key.
4.Attention matrix A = softmax(QKᵀ/√d_k) has shape (seq_len × seq_len) — A[i,j] = how much token i attends to token j.
5.Output = AV gives each token a context-aware representation.

Real Example

Scenario

Sentence: 'The animal didn't cross the street because it was too tired.' Self-attention lets 'it' attend strongly to 'animal' (weight 0.7) rather than 'street' (weight 0.1), resolving the pronoun reference.

What you would do

In Transformer & ML Foundations, apply Self Attention to this scenario: Sentence: 'The animal didn't cross the street because it was too tired. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Self Attention (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Self Attention happens in the code.

Self Attention

1import torch  # import dependencies2import torch.nn as nn  # import dependencies3
4class SelfAttention(nn.Module):  # define a data structure or component5    def __init__(self, d_model: int):  # define a reusable function6        super().__init__()7        self.W_q = nn.Linear(d_model, d_model)8        self.W_k = nn.Linear(d_model, d_model)9        self.W_v = nn.Linear(d_model, d_model)10
11    def forward(self, x, mask=None):  # define a reusable function12        Q, K, V = self.W_q(x), self.W_k(x), self.W_v(x)13        scores = torch.matmul(Q, K.transpose(-2, -1)) / (Q.size(-1) ** 0.5)  # key line for Self Attention14        if mask is not None:15            scores = scores.masked_fill(mask == 0, float('-inf'))16        weights = torch.softmax(scores, dim=-1)  # key line for Self Attention17        return torch.matmul(weights, V)  # return the result18
19# Causal mask for GPT-style generation20seq_len = 521mask = torch.tril(torch.ones(seq_len, seq_len))

Cheat Sheet

Quick recap

quick ref

•Self-attn: Q,K,V from same X
•Causal mask: tril(ones(n,n))
•Output = softmax(QKᵀ/√d)V
•Residual: x + sublayer(x)
•Pre-norm vs post-norm variants
•Parallel over sequence length

Common Mistakes

✕Omitting causal mask in decoder — model cheats during training
✕Confusing self-attention with cross-attention
✕Assuming attention weights are interpretable ground truth
✕Forgetting residual connections around attention blocks