Phase 1.1 · OptionalOptionalModule 5 of 21

Loss Functions

Training needs a single scalar objective to minimize. Loss functions map prediction errors to a number: lower loss = better model. Different tasks need different losses — regression, classification, and language modeling each have specialized formulations.

A report card for your model. MSE is like grading on distance from the correct answer. Cross-entropy is like grading how surprised the model was by the correct answer — heavily penalizing confident wrong predictions.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Loss functions quantify how wrong a model's predictions are — providing the signal that gradient descent uses to improve weights during training. MSE (Mean Squared Error): average (y - ŷ)² — for regression.
2.Cross-Entropy Loss: -Σ y·log(ŷ) — for classification; penalizes confident wrong predictions heavily. Binary CE for two classes; Categorical CE for multi-class.
3.Language modeling uses Cross-Entropy over vocabulary: loss = -log P(correct_next_token). Lower perplexity = lower CE loss.
4.Huber loss combines MSE and MAE — robust to outliers. Loss choice affects gradient behavior and what the model optimizes for.

Real Example

Scenario

LLM predicts next token. Vocabulary = 50,000 tokens. Model outputs 50,000 logits; softmax converts to probabilities. Correct token 'Paris' had probability 0.3 → loss = -log(0.3) ≈ 1.2. If model was confident wrong (p=0.01) → loss = -log(0.01) ≈ 4.6.

What you would do

In Transformer & ML Foundations, apply Loss Functions to this scenario: LLM predicts next token. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Loss Functions (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Loss Functions happens in the code.

Loss Functions

1import torch  # import dependencies2import torch.nn as nn  # import dependencies3
4# Classification5logits = torch.tensor([[2.0, 1.0, 0.1]])6target = torch.tensor([0])  # class 07ce_loss = nn.CrossEntropyLoss()(logits, target)  # key line for Loss Functions8
9# Language modeling (next token)10vocab_logits = torch.randn(1, 50257)  # GPT-2 vocab11next_token = torch.tensor([1234])12lm_loss = nn.CrossEntropyLoss()(vocab_logits, next_token)  # key line for Loss Functions13
14# Regression15pred = torch.tensor([3.5])16true = torch.tensor([4.0])17mse = nn.MSELoss()(pred, true)  # key line for Loss Functions18
19print(f"CE: {ce_loss:.3f}, LM: {lm_loss:.3f}, MSE: {mse:.3f}")  # show output for debugging

Cheat Sheet

Quick recap

quick ref

•MSE = mean((y - ŷ)²)
•CE = -Σ y·log(ŷ)
•LM loss = CE over vocab logits
•Perplexity = exp(CE)
•nn.CrossEntropyLoss() includes softmax
•Lower perplexity = better language model

Common Mistakes

✕Applying softmax before nn.CrossEntropyLoss (it includes softmax)
✕Using MSE for classification — wrong gradient behavior
✕Ignoring class imbalance — need weighted CE or focal loss
✕Confusing loss with accuracy — low loss doesn't always mean high accuracy