BM25
Embeddings smooth over rare tokens — 'CVE-2024-1234' and 'JIRA-4521' get lost in semantic space. BM25 excels at exact term matching, rare word sensitivity, and interpretability. It's fast, well-understood, and requires no GPU.
BM25 is like a library index card system — it finds books containing the exact words you searched for, weighted by how rare and important those words are in the collection.
Visual Workflows
Start here — scroll inside each diagram frame to explore, then use + / − to zoom up to 200% if needed.
Overview
Scroll inside the frame to explore · use + / − to zoom up to 200%
Scroll inside the frame to explore · use + / − to zoom up to 200%
Sparse retrieval
Scroll inside the frame to explore · use + / − to zoom up to 200%
Tokenize query, look up inverted index, score BM25, take top-k.
Where it belongs
Scroll inside the frame to explore · use + / − to zoom up to 200%
Not a replacement for embeddings — a teammate.
Key Takeaways
- 1.BM25 (Best Matching 25) is a probabilistic keyword ranking algorithm that scores documents by term frequency and inverse document frequency — the sparse retrieval backbone of hybrid search systems.
- 2.BM25 score = Σ IDF(qi) × (f(qi,D) × (k1+1)) / (f(qi,D) + k1×(1-b+b×|D|/avgdl)).
- 3.Parameters: k1 (term saturation, default 1.2), b (length normalization, default 0.75).
- 4.IDF downweights common terms ('the', 'is') and boosts rare terms.
- 5.Implementations: Elasticsearch, OpenSearch, rank_bm25 (Python), and built into Weaviate/Qdrant.
Real Example
Scenario
Query 'PTO policy section 4.2' — BM25 matches exact section reference in the employee handbook. Vector search might return general PTO content but miss the specific section. BM25 pinpoints it.
What you would do
In RAG Engineering, apply BM25 to this scenario: Query 'PTO policy section 4. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.
Practice Task
Open the Code Walkthrough below and run it locally. Change one parameter related to BM25 (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.
Code Walkthrough
Highlighted lines show where BM25 happens in the code.
1from rank_bm25 import BM25Okapi # import dependencies2
3corpus = [doc.page_content for doc in chunks]4tokenized = [doc.lower().split() for doc in corpus]5bm25 = BM25Okapi(tokenized) # key line for BM256
7query_tokens = "PTO policy section 4.2".lower().split()8scores = bm25.get_scores(query_tokens) # key line for BM259top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:10]10results = [chunks[i] for i in top_indices]Commands
Commands to Remember
from rank_bm25 import BM25Okapi # classic sparse retrieverbm25.get_scores(query_tokens) # one score per documentBM25 does not understand synonyms — pair it with embeddingsTokenize the same way at index time and query time
Cheat Sheet
Quick recap
quick ref- •BM25 = sparse/keyword retrieval
- •IDF boosts rare terms
- •k1=1.2, b=0.75
- •Exact match > semantic for IDs
- •Hybrid with vector via RRF
- •Elasticsearch for production scale
Common Mistakes
- ✕Aggressive stemming breaking product codes and IDs
- ✕Not updating BM25 index when documents change
- ✕Using BM25 alone for paraphrase-heavy queries
- ✕Inconsistent tokenization between index and query
- ✕Ignoring BM25 in favor of pure vector search