0
0Document Loaders1Chunking2Embedding Models3Vector Databases4Metadata5Retrievers6Hybrid Search7BM258Cross Encoder9Re-ranking10Query Expansion11Compression12Caching13RAG14LangChain Basics15ChromaDB16Streamlit17RAG Evaluation
RAG Engineering
Phase 3Module 8 of 18

BM25

Embeddings smooth over rare tokens — 'CVE-2024-1234' and 'JIRA-4521' get lost in semantic space. BM25 excels at exact term matching, rare word sensitivity, and interpretability. It's fast, well-understood, and requires no GPU.

BM25 is like a library index card system — it finds books containing the exact words you searched for, weighted by how rare and important those words are in the collection.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.BM25 (Best Matching 25) is a probabilistic keyword ranking algorithm that scores documents by term frequency and inverse document frequency — the sparse retrieval backbone of hybrid search systems.
  • 2.BM25 score = Σ IDF(qi) × (f(qi,D) × (k1+1)) / (f(qi,D) + k1×(1-b+b×|D|/avgdl)).
  • 3.Parameters: k1 (term saturation, default 1.2), b (length normalization, default 0.75).
  • 4.IDF downweights common terms ('the', 'is') and boosts rare terms.
  • 5.Implementations: Elasticsearch, OpenSearch, rank_bm25 (Python), and built into Weaviate/Qdrant.

Real Example

Scenario

Query 'PTO policy section 4.2' — BM25 matches exact section reference in the employee handbook. Vector search might return general PTO content but miss the specific section. BM25 pinpoints it.

What you would do

In RAG Engineering, apply BM25 to this scenario: Query 'PTO policy section 4. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to BM25 (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where BM25 happens in the code.

BM25
1from rank_bm25 import BM25Okapi  # import dependencies2
3corpus = [doc.page_content for doc in chunks]4tokenized = [doc.lower().split() for doc in corpus]5bm25 = BM25Okapi(tokenized)  # key line for BM256
7query_tokens = "PTO policy section 4.2".lower().split()8scores = bm25.get_scores(query_tokens)  # key line for BM259top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:10]10results = [chunks[i] for i in top_indices]

Cheat Sheet

Quick recap

quick ref
  • •BM25 = sparse/keyword retrieval
  • •IDF boosts rare terms
  • •k1=1.2, b=0.75
  • •Exact match > semantic for IDs
  • •Hybrid with vector via RRF
  • •Elasticsearch for production scale

Common Mistakes

  • ✕Aggressive stemming breaking product codes and IDs
  • ✕Not updating BM25 index when documents change
  • ✕Using BM25 alone for paraphrase-heavy queries
  • ✕Inconsistent tokenization between index and query
  • ✕Ignoring BM25 in favor of pure vector search