Phase 1.1 · OptionalOptionalModule 15 of 21

BERT

Before BERT, NLP models were either unidirectional (GPT) or shallow (Word2Vec). BERT reads text bidirectionally and produces context-dependent representations — enabling state-of-the-art fine-tuning on classification, NER, QA with minimal task-specific architecture.

Word2Vec gives each word a fixed dictionary definition. BERT is like a skilled reader who understands each word based on the full paragraph — 'bank' means river edge or financial institution depending on context.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only model pre-trained with masked language modeling — producing deep contextual embeddings that revolutionized NLP understanding tasks. Pre-training objectives: (1) Masked Language Model (MLM) — randomly mask 15% of tokens, predict them using bidirectional context.
2.(2) Next Sentence Prediction (NSP) — is sentence B after A? Fine-tuning: add task-specific head on [CLS] or token outputs.
3.Architecture: encoder-only, 12 layers (base) or 24 (large), 768/1024 dim. Input: [CLS] sent1 [SEP] sent2 [SEP].
4.Variants: RoBERTa (no NSP, more data), DistilBERT (smaller, faster), ALBERT (parameter sharing). BERT embeddings power semantic search, classification, and as encoder in early RAG systems.

Real Example

Scenario

Fine-tuning BERT for sentiment: input '[CLS] This movie was fantastic [SEP]' → 12 encoder layers → take [CLS] hidden state (768-dim) → linear layer → 2 classes (positive/negative). Achieves 95%+ on SST-2 with minimal training.

What you would do

In Transformer & ML Foundations, apply BERT to this scenario: Fine-tuning BERT for sentiment: input '[CLS] This movie was fantastic [SEP]' → 12 encoder layers → take [CLS] hidden state (768-dim) → linear layer → 2 classes (positive/negative). Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to BERT (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where BERT happens in the code.

BERT

1from transformers import BertForSequenceClassification, BertTokenizer, Trainer  # import dependencies2from datasets import load_dataset  # import dependencies3
4model = BertForSequenceClassification.from_pretrained(  # key line for BERT5    "bert-base-uncased", num_labels=2  # key line for BERT6)7tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")  # key line for BERT8
9dataset = load_dataset("imdb", split="train[:1000]")10def tokenize(ex):  # define a reusable function11    return tokenizer(ex["text"], truncation=True, padding="max_length", max_length=128)  # return the result12tokenized = dataset.map(tokenize, batched=True)13
14trainer = Trainer(model=model, train_dataset=tokenized, args=TrainingArguments(output_dir="./bert", num_train_epochs=1))  # key line for BERT15# trainer.train()

Cheat Sheet

Quick recap

quick ref

•BERT: encoder-only, bidirectional
•MLM: mask 15%, predict
•[CLS] for classification
•bert-base: 110M params
•Fine-tune: add task head, train
•Not autoregressive — no generation

Common Mistakes

✕Using BERT for autoregressive text generation
✕Not using [CLS] for sequence classification tasks
✕Forgetting segment embeddings for sentence-pair tasks
✕Confusing BERT pre-training with fine-tuning objectives