0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 12 of 21

Positional Encoding

Self-attention is permutation-invariant: shuffling input tokens gives the same attention patterns (just reordered). Without position information, the model cannot distinguish word order — catastrophic for language.

A bag of words vs a sentence. Attention alone treats input like a bag — positional encoding adds seat numbers so the model knows who sits where in the sentence.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Positional encoding injects information about token order into transformer models — which lack the inherent sequential processing of RNNs and need to know that 'cat sat' differs from 'sat cat'. Original Transformer: sinusoidal PE — PE(pos, 2i) = sin(pos/10000^(2i/d)), PE(pos, 2i+1) = cos(...).
  • 2.Added to token embeddings before first layer. Fixed, not learned.
  • 3.RoPE (modern): encodes position in attention by rotating Q and K vectors — relative position emerges naturally. Learned embeddings: trainable position vectors (like token embeddings).
  • 4.ALiBi: attention bias based on distance — no explicit encoding. GPT uses learned positional embeddings; LLaMA/Mistral use RoPE.

Real Example

Scenario

Without PE: 'dog bites man' and 'man bites dog' produce identical attention patterns. With PE: position 0's encoding differs from position 2's, so the model learns subject-verb-object structure.

What you would do

In Transformer & ML Foundations, apply Positional Encoding to this scenario: Without PE: 'dog bites man' and 'man bites dog' produce identical attention patterns. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Positional Encoding (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Positional Encoding happens in the code.

Positional Encoding
1import torch  # import dependencies2import math  # import dependencies3
4def sinusoidal_pe(seq_len: int, d_model: int) -> torch.Tensor:  # define a reusable function5    pe = torch.zeros(seq_len, d_model)6    position = torch.arange(seq_len).unsqueeze(1).float()7    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))8    pe[:, 0::2] = torch.sin(position * div_term)9    pe[:, 1::2] = torch.cos(position * div_term)10    return pe  # return the result11
12pe = sinusoidal_pe(128, 512)13token_emb = torch.randn(128, 512)14input_emb = token_emb + pe  # element-wise addition

Cheat Sheet

Quick recap

quick ref
  • •input = token_emb + pos_emb
  • •Sinusoidal: sin/cos of position
  • •Learned: nn.Embedding(max_len, d)
  • •RoPE: rotate Q,K by position
  • •ALiBi: linear attention bias
  • •Permutation-invariant without PE

Common Mistakes

  • ✕Forgetting positional encoding entirely in custom transformers
  • ✕Using learned PE beyond max_position_embeddings — out of range
  • ✕Confusing RoPE (applied in attention) with additive PE
  • ✕Assuming all modern LLMs use the same PE method