0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 14 of 21

Word2Vec

One-hot encoding gives sparse, high-dimensional, semantically meaningless vectors. Word2Vec learns that words appearing in similar contexts have similar meanings, producing compact embeddings that capture semantic relationships.

Plotting cities on a map: Paris and London are close (both European capitals), far from Tokyo. Word2Vec creates a 'semantic map' where similar words cluster together in vector space.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Word2Vec learns dense vector representations of words where semantic similarity corresponds to geometric proximity — 'king - man + woman ≈ queen' — laying groundwork for modern embeddings. Two architectures: CBOW (predict word from context) and Skip-gram (predict context from word).
  • 2.Training: sliding window over corpus, maximize P(context|word) via negative sampling (contrast with random non-context words). Result: embedding matrix where each word maps to a dense vector (typically 100-300 dims).
  • 3.Properties: linear analogies (king-man+woman≈queen), clustering of synonyms. Limitations: one vector per word (no context sensitivity — 'bank' river vs money), no subword handling.
  • 4.Superseded by contextual embeddings (ELMo, BERT) but intuition remains foundational.

Real Example

Scenario

Training on 'The king ruled the kingdom': Skip-gram with window=2 learns 'king' embedding predicts 'The', 'ruled'. After training, vector('king') - vector('man') + vector('woman') ≈ vector('queen').

What you would do

In Transformer & ML Foundations, apply Word2Vec to this scenario: Training on 'The king ruled the kingdom': Skip-gram with window=2 learns 'king' embedding predicts 'The', 'ruled'. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Word2Vec (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Word2Vec happens in the code.

Word2Vec
1from gensim.models import Word2Vec  # import dependencies2
3sentences = [4    ["king", "rules", "the", "kingdom"],5    ["queen", "rules", "the", "kingdom"],6    ["man", "works", "in", "the", "field"],7    ["woman", "works", "in", "the", "field"],8]9
10model = Word2Vec(sentences, vector_size=100, window=2, min_count=1, epochs=100)  # key line for Word2Vec11
12# Semantic similarity13print(model.wv.similarity("king", "queen"))  # show output for debugging14
15# Linear analogy16result = model.wv.most_similar(17    positive=["king", "woman"], negative=["man"], topn=118)19print(result)  # show output for debugging

Cheat Sheet

Quick recap

quick ref
  • •Skip-gram: word → context
  • •CBOW: context → word
  • •Negative sampling: k random negatives
  • •window=5 typical
  • •Static: one vec per word
  • •gensim Word2Vec API

Common Mistakes

  • ✕Using Word2Vec for polysemous words without context disambiguation
  • ✕Expecting one embedding to capture all word senses
  • ✕Skipping negative sampling — training becomes impractically slow
  • ✕Confusing Word2Vec embeddings with transformer token embeddings