Agentic AI Notebook
Back to Projects
ProductionPhase 8 60 hours(broken down below)

Production AI Platform

Full observability stack for AI applications — tracing, evaluation, prompt versioning, and cost monitoring.

PythonLangSmithPrometheusGrafanaKubernetes

Project walkthrough

Production AI Platform

1 / 6

Project Goal

Production observability platform for LLM applications at scale.

  • →Distributed tracing across all LLM calls
  • →Versioned prompt registry with rollback
  • →Cost, latency, and eval dashboards
  • →A/B testing and Kubernetes auto-scaling

Use ← → arrow keys or buttons to navigate the walkthrough

Time breakdown (60h)

Each phase maps to the estimated hours — follow in order for a realistic build schedule.

Distributed tracing

12h
  • •Instrument all LLM calls with LangSmith spans
  • •Propagate trace_id across microservices
  • •Trace viewer integration in admin UI

Prompt versioning

10h
  • •Postgres-backed prompt registry with semver tags
  • •Hot-reload prompts without redeploy
  • •Diff view and one-click rollback

Cost & eval dashboards

12h
  • •Custom Prometheus metrics: tokens, cost, latency p95
  • •Grafana panels for per-model cost breakdown
  • •Nightly eval job writing scores to Prometheus

A/B testing framework

10h
  • •Traffic splitter by user_id hash
  • •Statistical significance calculator for eval metrics
  • •Auto-promote winner after threshold

K8s deployment & auto-scaling

16h
  • •Helm charts for all platform components
  • •HPA on CPU and custom queue_depth metric
  • •Alerting rules for cost spikes and latency SLO breaches

Architecture

LLM services emit traces and custom metrics to LangSmith and Prometheus. A prompt registry stores versioned templates with rollback capability, while an A/B router splits traffic to compare versions. Grafana dashboards aggregate cost, latency, and eval scores, and HPA scales pods on request queue depth.

100%
Loading diagram...

Scroll inside the frame to explore · use + / − to zoom up to 200%

Prerequisites

  • Production Python service experience (FastAPI or similar)
  • LangSmith or OpenTelemetry tracing fundamentals
  • Prometheus + Grafana dashboard basics
  • Kubernetes deployment with Helm
  • Understanding of LLM cost drivers (tokens, model tier)

Setup steps

  1. Deploy a sample LLM microservice with LangSmith tracing enabled
  2. Install Prometheus operator and Grafana in a local K8s cluster (minikube)
  3. Create custom metrics: token_count, latency_histogram, cost_usd
  4. Set up prompt registry with versioned templates in Postgres
  5. Configure A/B test router splitting traffic between prompt versions
  6. Build Grafana dashboards for cost, latency, and eval scores

Features to build

  • Distributed tracing
  • Prompt versioning
  • Cost dashboards
  • A/B testing
  • Auto-scaling

Expected result

Show a live Grafana dashboard tracking 1M+ monthly LLM calls with per-model cost breakdown, p95 latency, eval faithfulness scores, and an active A/B test comparing two prompt versions with statistical significance.

Resume bullet points

  • →Built production AI observability platform monitoring 1M+ LLM calls/month
  • →Reduced inference costs by 45% through caching and model routing

Interview questions

How do you monitor LLM costs in production?
Emit per-request token counts and model ID to Prometheus, aggregate cost_usd by endpoint and tenant, set budget alerts, and use caching/routing to reduce spend on repeated or simple queries.
Why version prompts separately from code?
Prompts change far more often than application logic. A registry enables hot-reload, A/B testing, rollback without redeploy, and audit trails for compliance.
What metrics matter most for LLM SLOs?
Latency p95/p99, error rate, token cost per request, eval faithfulness score, and cache hit rate. Alert on SLO breaches, not every anomaly.