Agentic AI Notebook
Back to Projects
ProductionPhase 4 35 hours(broken down below)

RAG Eval CI Pipeline

RAG application with golden evaluation datasets integrated into CI/CD — block merges when retrieval or answer quality regresses.

PythonLangChainRAGASGitHub ActionsPinecone

Project walkthrough

RAG Eval CI Pipeline

1 / 6

Project Goal

Treat RAG quality like code coverage — test it in CI.

  • →Golden dataset of 100 Q&A pairs with gold answers
  • →RAGAS metrics on every pull request
  • →Block merges when quality regresses
  • →Track metric trends over time

Use ← → arrow keys or buttons to navigate the walkthrough

Time breakdown (35h)

Each phase maps to the estimated hours — follow in order for a realistic build schedule.

RAG pipeline setup

8h
  • •Configurable chunk size, overlap, and top-k via env vars
  • •Ingestion script for eval corpus (50 docs)
  • •Baseline retrieval + generation chain

Golden eval dataset

6h
  • •Curate 100 Q&A pairs with gold answers and source citations
  • •Human review of 20% for label quality
  • •Version dataset in repo with changelog

CI/CD integration

8h
  • •GitHub Actions workflow: lint → eval → gate
  • •Eval script outputs JSON with per-metric scores
  • •PR comment bot posting eval results table

Metrics & thresholds

7h
  • •RAGAS: faithfulness, answer_relevancy, context_precision
  • •Configurable thresholds in eval config YAML
  • •Per-question failure report for debugging

Regression alerts & docs

6h
  • •Nightly eval on main, store metrics in SQLite/Postgres
  • •Alert when 7-day rolling average drops 5%
  • •README documenting how to add eval cases and tune thresholds

Architecture

A RAG application serves production queries while a parallel eval runner executes the golden dataset on every PR. RAGAS computes faithfulness, answer relevance, and context precision; CI blocks merges when any metric drops below thresholds, and nightly runs on main detect slow regressions.

100%
Loading diagram...

Scroll inside the frame to explore · use + / − to zoom up to 200%

Prerequisites

  • Working RAG pipeline (ingestion, retrieval, generation)
  • RAGAS or DeepEval library familiarity
  • GitHub Actions CI/CD experience
  • Understanding of faithfulness, relevance, and context precision metrics
  • Pinecone or ChromaDB vector store

Setup steps

  1. Build or reuse a RAG app with configurable chunk size and top-k
  2. Create golden eval set: 100 Q&A pairs with expected answers and source docs
  3. Install RAGAS and write eval script outputting JSON metrics
  4. Add GitHub Actions workflow running eval on every PR
  5. Set quality gates: faithfulness ≥ 0.85, context precision ≥ 0.75
  6. Configure Slack/email alert on main branch regression

Features to build

  • Golden eval dataset
  • CI/CD quality gates
  • RAGAS metrics
  • Regression alerts
  • Chunking A/B comparison

Expected result

Open a PR that changes chunk size, watch CI run the 100-case eval suite, see RAGAS scores posted as a PR comment, and demonstrate a merge blocked because faithfulness dropped below 0.85.

Resume bullet points

  • →Integrated RAG evaluation into CI/CD blocking merges on quality regression
  • →Built golden dataset of 100 Q&A pairs with automated RAGAS scoring

Interview questions

Which RAGAS metrics should gate a CI merge?
Faithfulness (is the answer grounded in retrieved context?) is the most critical gate. Add context_precision (are retrieved chunks relevant?) and answer_relevancy (does it address the question?). Set thresholds based on baseline + acceptable regression margin.
How do you build a golden eval dataset for RAG?
Curate real user questions, have domain experts write gold answers with source citations, cover edge cases (no answer in docs, ambiguous queries), version the dataset in git, and audit 20% of labels regularly.
What if CI eval is too slow for every PR?
Run full 100-case suite nightly and a 20-case smoke subset on every PR. Cache embeddings, parallelize eval runs, and use smaller models for RAGAS judge where possible.