Agentic AI Notebook
Back to Projects
IntermediatePhase 4 20 hours(broken down below)

Enterprise Chatbot

Production-grade RAG chatbot with hybrid search, re-ranking, and evaluation metrics for enterprise knowledge bases.

PythonLangChainPineconeFastAPIReact

Project walkthrough

Enterprise Chatbot

1 / 6

Project Goal

Enterprise RAG with hybrid search, reranking, and measurable quality.

  • →Ingest 10K+ documents with ACL metadata
  • →Hybrid BM25 + vector retrieval with reranking
  • →Citation-enforced answers
  • →Evaluation dashboard with regression tracking

Use ← → arrow keys or buttons to navigate the walkthrough

Time breakdown (20h)

Each phase maps to the estimated hours — follow in order for a realistic build schedule.

Ingestion pipeline

4h
  • •Async worker for PDF/Markdown/HTML parsing
  • •Chunk with metadata: source, page, ACL tags
  • •Dual-write to Pinecone and BM25 index

Hybrid search

5h
  • •Reciprocal rank fusion of BM25 + vector results
  • •Metadata filters for tenant and document type
  • •Configurable top-k and score thresholds

Re-ranking & citations

5h
  • •Cross-encoder rerank top-20 to top-5
  • •Prompt with citation format [source:page]
  • •Post-validator rejects uncited claims

Evaluation dashboard

4h
  • •Run golden Q&A set nightly
  • •Track precision@5, faithfulness, latency
  • •Regression alerts on metric drops

API & React UI

2h
  • •Streaming SSE chat endpoint
  • •Citation sidebar with source previews
  • •Admin panel for re-indexing

Architecture

Documents are ingested with metadata (source, ACL, timestamp) into Pinecone while a parallel BM25 index enables keyword recall. Queries run hybrid retrieval, cross-encoder re-ranking narrows to top passages, and the LLM generates answers with mandatory inline citations validated post-generation.

100%
Loading diagram...

Scroll inside the frame to explore · use + / − to zoom up to 200%

Prerequisites

  • Completed a basic RAG project (PDF chat or equivalent)
  • Pinecone account and API key
  • Understanding of BM25 vs dense retrieval tradeoffs
  • FastAPI + React full-stack experience
  • Familiarity with RAG evaluation metrics (precision, faithfulness)

Setup steps

  1. Provision Pinecone index with 1536-dim vectors and metadata filters
  2. Set up FastAPI backend with document ingestion worker
  3. Install rank_bm25, sentence-transformers for cross-encoder reranking
  4. Scaffold React chat UI with citation sidebar
  5. Prepare 50+ enterprise docs (wikis, policies) for ingestion
  6. Create a golden eval set of 30 Q&A pairs with expected citations

Features to build

  • Hybrid search (BM25 + vector)
  • Cross-encoder re-ranking
  • Citation tracking
  • Evaluation dashboard

Expected result

Demo querying a 10K-chunk knowledge base with hybrid search, showing reranked citations in the answer sidebar, and an eval dashboard reporting precision@5 above 0.8 on your golden set.

Resume bullet points

  • →Architected enterprise RAG chatbot serving 10K+ documents with hybrid search
  • →Implemented re-ranking pipeline improving retrieval precision by 35%

Interview questions

Why use hybrid search instead of vectors alone?
Vectors miss exact keyword matches (SKUs, error codes, acronyms) while BM25 misses semantic paraphrases. Reciprocal rank fusion combines both recall profiles for enterprise docs with mixed content types.
How do you evaluate a RAG chatbot in production?
Maintain a golden Q&A set with expected citations, track precision@k, answer faithfulness (LLM-judge or human), latency p95, and citation accuracy. Run nightly regression and alert on drops.
What does the cross-encoder reranker add over bi-encoder retrieval?
Bi-encoders embed query and doc separately (fast but shallow). Cross-encoders score query-doc pairs jointly (slower but more accurate), ideal for reranking top-20 candidates before LLM generation.