Metadata
Pure semantic search returns the most similar text regardless of whether it's from the right document, tenant, time period, or access level. Metadata filters narrow the search space and ensure retrieved chunks are relevant, authorized, and citable.
Metadata is like the labels on filing cabinet folders — without them, you might find a similar document, but from the wrong year, wrong department, or wrong client's folder.
Visual Workflows
Start here — study each diagram, then use + / − to zoom if needed.
Scroll inside the frame · use + / − to zoom
Key Takeaways
- 1.Metadata is structured information attached to each chunk — enabling filtered retrieval, source citations, access control, and debugging — turning a dumb vector search into a precise, context-aware retrieval system.
- 2.Essential metadata fields: source_uri, doc_id, chunk_index, title, section, page, created_at, updated_at, tenant_id, user_acl, language, content_type.
- 3.Use metadata for: pre-filtering (search only 2024 policies), post-filtering (re-rank by recency), citation generation (page + section in answers), and debugging (which doc failed retrieval).
- 4.Design a metadata schema upfront — retrofitting is painful.
- 5.Index metadata fields in your vector DB for fast filtering.
Real Example
Scenario
User asks 'What changed in the Q3 security policy?' System filters metadata {doc_type: 'policy', department: 'security', updated_at: > '2024-07-01'} before vector search, ensuring only recent security policies are retrieved.
What you would do
In RAG Engineering, apply Metadata to this scenario: User asks 'What changed in the Q3 security policy?' System filters metadata {doc_type: 'policy', department: 'security', updated_at: > '2024-07-01'} before vector search, ensuring only recent security policies are retrieved. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.
Practice Task
Open the Code Walkthrough below and run it locally. Change one parameter related to Metadata (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.
Code Walkthrough
Highlighted lines show where Metadata happens in the code.
1from qdrant_client.models import Filter, FieldCondition, MatchValue, Range # import dependencies2
3tenant_filter = Filter(must=[4 FieldCondition(key="tenant_id", match=MatchValue(value="acme")),5 FieldCondition(key="doc_type", match=MatchValue(value="policy")),6 FieldCondition(key="updated_at", range=Range(gte="2024-07-01")),7])8
9# Attach metadata at chunk time10chunk.metadata.update({ # key line for Metadata11 "doc_id": "pol-2024-q3",12 "section": "Access Control",13 "page": 14,14 "tenant_id": "acme",15 "doc_type": "policy",16 "updated_at": "2024-09-15",17})Cheat Sheet
Quick recap
quick ref- •tenant_id filter = mandatory in SaaS
- •source + page = citations
- •Pre-filter ACLs at DB level
- •Schema design before ingestion
- •updated_at for recency filtering
- •chunk_index for ordering
Common Mistakes
- ✕No tenant_id — cross-tenant data leakage
- ✕Metadata schema designed after ingestion started
- ✕Storing full document text in metadata instead of payload
- ✕No citation fields — unverifiable answers
- ✕App-layer filtering instead of DB-level filters