Phase 1.1 · OptionalOptionalModule 2 of 21

Gradient Descent

Neural networks have millions of parameters — no closed-form solution exists. Gradient descent provides a general, scalable method to minimize loss by using calculus (derivatives) to know which direction to adjust each weight.

You're blindfolded on a hilly landscape trying to reach the lowest valley. You feel the slope under your feet and take small steps downhill. Learning rate controls step size — too big and you overshoot; too small and you take forever.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Gradient descent is the optimization algorithm that finds good neural network weights by repeatedly stepping downhill on the loss surface in the direction of steepest decrease. Loss L is a function of all weights θ.
2.The gradient ∇L tells you the direction of steepest increase. Update rule: θ_new = θ_old - η·∇L, where η is learning rate.
3.Mini-batch GD computes gradient on a subset of data (faster, noisy but effective). Stochastic GD uses one sample.
4.In practice, mini-batch (32-512) balances speed and stability. The loss surface is non-convex for neural nets — GD finds local minima that generalize well.

Real Example

Scenario

Training a linear regression: loss = MSE. Gradient of MSE w.r.t. weight w points uphill. Subtracting η·gradient moves w toward the value that minimizes average squared error across training points.

What you would do

In Transformer & ML Foundations, apply Gradient Descent to this scenario: Training a linear regression: loss = MSE. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Gradient Descent (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Gradient Descent happens in the code.

Gradient Descent

1import torch  # import dependencies2
3# Simple gradient descent on y = 2x4w = torch.tensor([0.0], requires_grad=True)5lr = 0.16
7for step in range(50):8    # Forward: predict y = w * x9    x = torch.tensor([3.0])10    y_pred = w * x11    loss = (y_pred - 6.0) ** 2  # target y=612
13    loss.backward()              # compute gradient14    with torch.no_grad():15        w -= lr * w.grad         # update weight16    w.grad.zero_()17
18print(f"Learned w: {w.item():.2f}")  # ~2.0

Cheat Sheet

Quick recap

quick ref

•θ ← θ - η·∇L
•η = learning rate (typical: 1e-4 to 1e-2)
•loss.backward() → gradients
•optimizer.step() → weight update
•Mini-batch size: 32, 64, 128, 256
•Watch loss curve for divergence

Common Mistakes

✕Using learning rate too high — loss explodes to NaN
✕Forgetting zero_grad() between iterations in PyTorch
✕Assuming GD always finds global minimum
✕Not monitoring loss curve during training