0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 6 of 21

Optimizers

Plain SGD is slow and sensitive to learning rate choice. Different parameters need different update speeds (embeddings vs attention weights). Optimizers like Adam adapt per-parameter learning rates and use momentum for faster, more stable convergence.

Basic GD is walking downhill blindly. Momentum is a ball rolling downhill — it accelerates in consistent directions and rolls over small bumps. Adam is a smart hiker with a GPS that remembers which paths were steep and adjusts step size per direction.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Optimizers are algorithms that use gradients to update neural network weights — going beyond basic gradient descent with momentum, adaptive learning rates, and memory of past gradients. SGD: θ -= lr·∇L.
  • 2.SGD+Momentum: accumulates velocity v = βv + ∇L; θ -= lr·v — faster convergence, smooths noisy gradients. Adam: maintains per-parameter first moment (mean of gradients) and second moment (variance).
  • 3.Update: θ -= lr·m̂/(√v̂ + ε). Adapts learning rate per parameter automatically.
  • 4.AdamW: Adam with decoupled weight decay (L2 regularization applied to weights, not gradients) — standard for transformer training. LLM fine-tuning often uses AdamW with lr=1e-5 to 5e-5, warmup, and cosine decay.

Real Example

Scenario

Fine-tuning BERT: AdamW optimizer, lr=2e-5, warmup 10% of steps, linear decay. Weight decay=0.01 regularizes. Momentum helps push through noisy mini-batch gradients from small datasets.

What you would do

In Transformer & ML Foundations, apply Optimizers to this scenario: Fine-tuning BERT: AdamW optimizer, lr=2e-5, warmup 10% of steps, linear decay. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Optimizers (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Optimizers happens in the code.

Optimizers
1import torch  # import dependencies2import torch.nn as nn  # import dependencies3
4model = nn.Linear(10, 2)5
6# AdamW — default for transformer fine-tuning7optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)8
9# Learning rate scheduler with warmup10from torch.optim.lr_scheduler import CosineAnnealingLR  # import dependencies11scheduler = CosineAnnealingLR(optimizer, T_max=100)12
13for step in range(100):14    optimizer.zero_grad()15    loss = model(torch.randn(8, 10)).sum()  # dummy16    loss.backward()17    optimizer.step()18    scheduler.step()

Cheat Sheet

Quick recap

quick ref
  • •AdamW(lr=2e-5, weight_decay=0.01)
  • •zero_grad → backward → step
  • •Warmup: linear lr increase first N steps
  • •Cosine decay: lr follows cosine curve
  • •Momentum β=0.9 typical
  • •SGD better generalization sometimes

Common Mistakes

  • ✕Using same learning rate for pre-training and fine-tuning
  • ✕Skipping warmup on large transformer fine-tuning runs
  • ✕Applying weight decay to bias and LayerNorm parameters
  • ✕Not using a learning rate scheduler for long training runs