0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 1 of 21

Neural Networks

Hand-crafted rules fail on messy real-world data (images, language, speech). Neural networks learn representations automatically from examples, scaling from simple classifiers to billion-parameter language models.

Imagine a factory assembly line where each worker tweaks a dial based on how far the final product missed the target. After thousands of iterations, the line produces correct outputs without anyone writing explicit rules.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Neural networks are layered functions that learn patterns from data by adjusting weights — the universal building block behind every modern AI system including LLMs. A neural network stacks layers of neurons.
  • 2.Each neuron computes weighted_sum(inputs) + bias, then applies a non-linear activation. Forward pass: input flows layer-by-layer to produce output.
  • 3.Training adjusts weights to minimize loss. Depth enables hierarchical features — early layers detect edges/words, deeper layers combine them into concepts.
  • 4.Parameters = all weights and biases; capacity grows with width (neurons per layer) and depth (number of layers).

Real Example

Scenario

Classifying emails as spam: 500 input features (word counts) → 128-neuron hidden layer → 1 output (spam probability). The network learns which word patterns correlate with spam without you defining rules.

What you would do

In Transformer & ML Foundations, apply Neural Networks to this scenario: Classifying emails as spam: 500 input features (word counts) → 128-neuron hidden layer → 1 output (spam probability). Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Neural Networks (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Neural Networks happens in the code.

Neural Networks
1import torch  # import dependencies2import torch.nn as nn  # import dependencies3
4class SimpleClassifier(nn.Module):  # define a data structure or component5    def __init__(self, input_dim: int, hidden_dim: int):  # define a reusable function6        super().__init__()7        self.net = nn.Sequential(8            nn.Linear(input_dim, hidden_dim),9            nn.ReLU(),10            nn.Linear(hidden_dim, 1),11            nn.Sigmoid(),12        )13
14    def forward(self, x: torch.Tensor) -> torch.Tensor:  # define a reusable function15        return self.net(x)  # return the result16
17model = SimpleClassifier(input_dim=500, hidden_dim=128)18x = torch.randn(32, 500)  # batch of 32 emails19output = model(x)           # shape: (32, 1)

Cheat Sheet

Quick recap

quick ref
  • •y = activation(Wx + b)
  • •Params = weights + biases (learned)
  • •Hyperparams = lr, layers, epochs (set manually)
  • •ReLU/GELU = common activations
  • •Forward pass = inference; backward = training
  • •nn.Module in PyTorch defines architecture

Common Mistakes

  • ✕Confusing parameters with hyperparameters in interviews
  • ✕Building networks without activation functions between layers
  • ✕Not normalizing inputs — causes slow or unstable training
  • ✕Judging model quality on training accuracy alone