0
0What is AI1ML vs DL vs GenAI2LLMs3Transformers4Tokens5Tokenization6Embeddings7Similarity Search8Prompt Engineering9Temperature10Top P11Context Window12Hallucination13Model Providers14Open vs Closed Models
Generative AI Foundations
Phase 1Module 4 of 15

Transformers

Previous architectures (RNNs) processed text sequentially — slow and struggled with long-range dependencies. Transformers process all tokens simultaneously via attention, enabling training on massive datasets and capturing long-range context.

Reading a book: RNNs read word-by-word left to right (slow, forget early chapters). Transformers read the whole page at once and highlight which words relate to each other (fast, sees connections across the entire text).

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Transformers are the neural network architecture powering all modern LLMs — using self-attention to process entire sequences in parallel rather than one token at a time.
  • 2.A transformer block has two sub-layers: multi-head self-attention and feed-forward network, each with residual connections and layer normalization.
  • 3.Self-attention computes how much each token should attend to every other token — capturing relationships like pronoun references and semantic similarity.
  • 4.Multi-head means multiple attention patterns in parallel.
  • 5.Stacking dozens of blocks creates deep representations.

Real Example

Scenario

In 'The cat sat on the mat because it was tired', self-attention lets 'it' strongly attend to 'cat' (not 'mat'), resolving the pronoun reference across the sentence.

What you would do

In Generative AI Foundations, apply Transformers to this scenario: In 'The cat sat on the mat because it was tired', self-attention lets 'it' strongly attend to 'cat' (not 'mat'), resolving the pronoun reference across the sentence. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Transformers (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Transformers happens in the code.

Transformers
1# Conceptual attention computation (simplified)2import torch  # import dependencies3import torch.nn.functional as F  # import dependencies4
5# Q, K, V = query, key, value projections of input6def self_attention(Q, K, V):  # define a reusable function7    scores = Q @ K.transpose(-2, -1) / (Q.size(-1) ** 0.5)8    weights = F.softmax(scores, dim=-1)9    return weights @ V  # return the result10
11seq_len, d_model = 10, 6412Q = K = V = torch.randn(1, seq_len, d_model)13output = self_attention(Q, K, V)  # key line for Transformers14print(output.shape)  # (1, 10, 64)

Cheat Sheet

Quick recap

quick ref
  • •Attention(Q,K,V) = softmax(QK^T/√d)V
  • •Multi-head attention
  • •GPT = decoder-only
  • •BERT = encoder-only
  • •Positional encoding
  • •Residual + LayerNorm

Common Mistakes

  • ✕Thinking you need to implement transformers to build AI apps
  • ✕Confusing encoder and decoder architectures
  • ✕Ignoring that attention is quadratic in sequence length
  • ✕Assuming more layers always means better for your use case