0
0Document Loaders1Chunking2Embedding Models3Vector Databases4Metadata5Retrievers6Hybrid Search7BM258Cross Encoder9Re-ranking10Query Expansion11Compression12Caching13RAG14LangChain Basics15ChromaDB16Streamlit17RAG Evaluation
RAG Engineering
Phase 3Module 13 of 18

Caching

RAG pipelines make 3-5 expensive calls per query: embed query, vector search, re-rank, LLM generate. Many queries repeat ('What is our refund policy?'). Without caching, you pay full cost every time. Caching can cut p95 latency by 80% and API costs by 50%+ for support and FAQ use cases.

Caching is like a restaurant keeping popular dishes pre-made — the first customer waits 20 minutes, but the next fifty get served in 2 minutes because the kitchen already has it ready.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Caching in RAG stores and reuses expensive computation results — query embeddings, retrieval results, and LLM responses — to reduce latency, cost, and load on downstream services. Cache layers: (1) Query embedding cache — hash query text → stored vector.
  • 2.(2) Retrieval cache — hash(query + filters) → top-k doc IDs. (3) Semantic cache — embed query, find similar cached queries above threshold (GPTCache).
  • 3.(4) LLM response cache — exact or semantic match returns stored answer. (5) Prompt cache (OpenAI/Anthropic) — cache static system prompt prefix.
  • 4.Invalidation: TTL-based, event-driven on document update, version-keyed on pipeline changes. Production: Redis for hot cache, cache hit rate monitoring, and graceful degradation on cache miss.

Real Example

Scenario

Support bot sees 'how to reset password' 500 times/day. Semantic cache (threshold 0.95) matches 85% to cached responses. Average latency drops from 2.1s to 180ms. Monthly LLM cost reduced 60%.

What you would do

In RAG Engineering, apply Caching to this scenario: Support bot sees 'how to reset password' 500 times/day. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Caching (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Caching happens in the code.

Caching
1import hashlib  # import dependencies2import json  # import dependencies3import redis  # import dependencies4
5cache = redis.Redis()6
7def cache_key(prefix: str, query: str, filters: dict) -> str:  # define a reusable function8    raw = f"{query}:{json.dumps(filters, sort_keys=True)}"9    return f"{prefix}:{hashlib.sha256(raw.encode()).hexdigest()}"  # return the result10
11def get_cached_retrieval(query: str, filters: dict) -> list | None:  # define a reusable function12    key = cache_key("retrieval", query, filters)13    data = cache.get(key)14    return json.loads(data) if data else None  # return the result15
16def set_cached_retrieval(query: str, filters: dict, results: list, ttl: int = 3600):  # define a reusable function17    key = cache_key("retrieval", query, filters)18    cache.setex(key, ttl, json.dumps(results))

Cheat Sheet

Quick recap

quick ref
  • •Cache at every expensive step
  • •Redis = hot cache
  • •Semantic threshold ≥ 0.95
  • •Invalidate on doc update
  • •Prompt cache: static first
  • •Monitor hit rate

Common Mistakes

  • ✕No cache invalidation — stale answers after doc updates
  • ✕Semantic cache threshold too low — wrong answers returned
  • ✕Caching before evaluation pipeline is stable
  • ✕Not monitoring cache hit rate
  • ✕Ignoring provider-level prompt caching discounts