0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 4 of 21

Activation Functions

Without activations, stacking layers is mathematically equivalent to one matrix multiply. Non-linear activations let networks approximate arbitrary functions (universal approximation) and learn hierarchical features.

A linear layer is like adjusting volume on a stereo — you can only make things louder or quieter. An activation function is the equalizer that shapes the sound — boosting bass, cutting treble — enabling rich, complex output.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Activation functions introduce non-linearity after each linear layer, enabling neural networks to learn complex patterns instead of collapsing into a single linear transform. Common activations: ReLU(x) = max(0,x) — fast, avoids vanishing gradients for positive values, but 'dead neurons' if always negative.
  • 2.GELU(x) ≈ x·Φ(x) — smooth, used in GPT/BERT transformers. Sigmoid σ(x) = 1/(1+e⁻ˣ) — outputs (0,1), used in gates and binary output.
  • 3.Tanh — zero-centered sigmoid, range (-1,1). Softmax — converts logits to probability distribution (sum=1), used at output layer for classification.
  • 4.LLMs use GELU/SwiGLU in FFN blocks; output uses softmax over vocabulary.

Real Example

Scenario

In a transformer FFN: input 768-dim → Linear to 3072 → GELU → Linear back to 768. GELU's smooth gating lets the model learn which dimensions to pass through, improving over ReLU in language tasks.

What you would do

In Transformer & ML Foundations, apply Activation Functions to this scenario: In a transformer FFN: input 768-dim → Linear to 3072 → GELU → Linear back to 768. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Activation Functions (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Activation Functions happens in the code.

Activation Functions
1import torch  # import dependencies2import torch.nn as nn  # import dependencies3import torch.nn.functional as F  # import dependencies4
5x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])6
7relu = F.relu(x)       # [0, 0, 0, 1, 2]8gelu = F.gelu(x)       # smooth, near 0 for negatives9sigmoid = torch.sigmoid(x)10
11# Transformer FFN pattern12ffn = nn.Sequential(13    nn.Linear(768, 3072),14    nn.GELU(),15    nn.Linear(3072, 768),16)

Cheat Sheet

Quick recap

quick ref
  • •ReLU = max(0, x)
  • •GELU = x · Φ(x) — GPT family
  • •SwiGLU = x · SiLU(Wx) — LLaMA
  • •Softmax = exp(xi)/Σexp(xj)
  • •Sigmoid for gates/binary output
  • •No activation = linear collapse

Common Mistakes

  • ✕Using softmax in hidden layers — causes saturation issues
  • ✕Applying sigmoid in deep hidden layers — vanishing gradients
  • ✕Forgetting softmax at multi-class output layer
  • ✕Confusing which activation modern LLMs actually use