0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 3 of 21

Backpropagation

A network with millions of weights needs millions of gradients per training step. Naively computing each gradient separately is O(n²). Backprop reuses intermediate computations via the chain rule, making training deep networks feasible in O(n) time.

A billing error at the end of a supply chain: backprop traces backward through each supplier to assign blame proportionally. Each layer asks 'how much did my output contribute to the final error?' and passes that signal backward.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Backpropagation is the algorithm that efficiently computes gradients of the loss with respect to every weight in a neural network using the chain rule of calculus.
  • 2.Forward pass stores activations at each layer.
  • 3.Backward pass applies chain rule from output to input: ∂L/∂w = ∂L/∂activation · ∂activation/∂z · ∂z/∂w.
  • 4.Each layer computes its local gradient and passes ∂L/∂input upstream.
  • 5.PyTorch's autograd automates this — you define forward(), call loss.backward(), gradients populate .grad attributes.

Real Example

Scenario

Two-layer network: loss depends on output of layer 2, which depends on layer 1 activations, which depend on layer 1 weights. Backprop computes ∂L/∂W2 first, then uses that to compute ∂L/∂W1 without re-deriving from scratch.

What you would do

In Transformer & ML Foundations, apply Backpropagation to this scenario: Two-layer network: loss depends on output of layer 2, which depends on layer 1 activations, which depend on layer 1 weights. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Backpropagation (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Backpropagation happens in the code.

Backpropagation
1import torch  # import dependencies2import torch.nn as nn  # import dependencies3
4model = nn.Sequential(5    nn.Linear(10, 5),6    nn.ReLU(),7    nn.Linear(5, 1),8)9
10x = torch.randn(4, 10)11y_true = torch.randn(4, 1)12
13# Forward14y_pred = model(x)15loss = nn.MSELoss()(y_pred, y_true)16
17# Backward — autograd computes all gradients18loss.backward()19
20# Inspect gradients21for name, param in model.named_parameters():22    print(f"{name}: grad shape {param.grad.shape}")  # show output for debugging

Cheat Sheet

Quick recap

quick ref
  • •Chain rule: ∂L/∂x = ∂L/∂y · ∂y/∂x
  • •loss.backward() → all .grad populated
  • •optimizer.zero_grad() before each step
  • •Vanishing: sigmoid in deep nets
  • •Fix: ReLU, residual connections, LayerNorm
  • •gradcheck validates custom gradients

Common Mistakes

  • ✕Forgetting optimizer.zero_grad() — gradients accumulate incorrectly
  • ✕Detaching tensors unintentionally, breaking the graph
  • ✕Assuming backprop finds global optimum — it only computes local gradients
  • ✕Not understanding that inference (eval mode) skips gradient computation