0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 10 of 21

Encoder

Understanding tasks (classification, NER, QA) need full context from both directions. The encoder builds deep contextualized representations where each token's embedding encodes information from the entire input sequence.

A team reviewing a document together — everyone reads the full report and discusses it. Each team member's understanding is enriched by seeing the whole picture, not just what came before.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.The transformer encoder processes input sequences with bidirectional self-attention — each token can see all other tokens, producing rich contextual representations for understanding tasks. Encoder stack: N identical layers, each containing (1) Multi-Head Self-Attention (bidirectional, no causal mask), (2) Add & Norm, (3) Position-wise FFN (Linear → GELU → Linear), (4) Add & Norm.
  • 2.Input: token embeddings + positional encoding. Output: contextualized representations for each token.
  • 3.BERT uses encoder-only architecture. Each layer refines representations — lower layers capture syntax, higher layers capture semantics.
  • 4.Encoder output feeds cross-attention in decoder (seq2seq) or classification head (BERT).

Real Example

Scenario

BERT encoding 'Time flies like an arrow': after 12 encoder layers, the embedding for 'flies' encodes that it's a verb (not insects) because bidirectional attention saw 'arrow' and 'Time' on both sides.

What you would do

In Transformer & ML Foundations, apply Encoder to this scenario: BERT encoding 'Time flies like an arrow': after 12 encoder layers, the embedding for 'flies' encodes that it's a verb (not insects) because bidirectional attention saw 'arrow' and 'Time' on both sides. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Encoder (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Encoder happens in the code.

Encoder
1from transformers import BertModel, BertTokenizer  # import dependencies2
3tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")4model = BertModel.from_pretrained("bert-base-uncased")5
6text = "Time flies like an arrow"7inputs = tokenizer(text, return_tensors="pt")8outputs = model(**inputs)9
10# outputs.last_hidden_state: (1, seq_len, 768)11# Each token has a contextualized 768-dim embedding12print(outputs.last_hidden_state.shape)  # show output for debugging

Cheat Sheet

Quick recap

quick ref
  • •Encoder = bidirectional MHA + FFN
  • •BERT: 12 layers, 768 dim, 12 heads
  • •[CLS] for classification
  • •No causal mask
  • •Add&Norm after each sublayer
  • •Output: (batch, seq_len, d_model)

Common Mistakes

  • ✕Applying causal mask in encoder — loses bidirectional context
  • ✕Using encoder for autoregressive generation without modification
  • ✕Ignoring [CLS] vs mean-pooling tradeoffs for classification
  • ✕Confusing encoder layers with decoder layers