ProductionPhase 9 40 hours(broken down below)

K8s Agent Deployment

Deploy AI agents on Kubernetes with Helm charts, horizontal pod autoscaling, health probes, and zero-downtime rollouts.

PythonKubernetesHelmDockerPrometheusLangGraph

Project walkthrough

1 / 6

Project Goal

Production-grade agent deployment on Kubernetes with autoscaling.

→Containerize LangGraph agent as a stateless service
→Helm charts for reproducible deploys
→HPA scaling on queue depth custom metric
→Zero-downtime rolling updates

Use ← → arrow keys or buttons to navigate the walkthrough

Time breakdown (40h)

Each phase maps to the estimated hours — follow in order for a realistic build schedule.

Agent containerization

•Multi-stage Dockerfile: builder + slim runtime
•Health endpoint: /health (liveness) and /ready (model loaded)
•Non-root user and read-only filesystem

Helm charts & manifests

•Helm chart with values.yaml for dev/staging/prod
•Deployment, Service, ConfigMap, Secret templates
•Resource requests/limits: 512Mi RAM, 500m CPU per pod

HPA & resource tuning

•Custom metric: agent_queue_depth via Prometheus adapter
•HPA: min 2, max 50 replicas, target queue depth 10
•Load test validating scale-up in <60s

Ingress & secrets

•Ingress with TLS termination
•K8s Secrets for OpenAI/Anthropic API keys
•External Secrets Operator for production key rotation

Monitoring & rollout

•Prometheus ServiceMonitor scraping agent metrics
•Grafana dashboard: queue depth, pod count, latency
•Rolling update with maxUnavailable=0, verify zero downtime

Architecture

Agent services run as Kubernetes Deployments behind a ClusterIP Service, with HPA scaling replicas based on queue depth.custom metrics. Secrets mount LLM API keys, ConfigMaps hold prompt templates, and rolling updates with readiness probes ensure zero-downtime deploys.

100%

Loading diagram...

Scroll inside the frame to explore · use + / − to zoom up to 200%

Prerequisites

Docker multi-stage builds for Python apps
Kubernetes fundamentals: Deployments, Services, ConfigMaps, Secrets
Helm chart templating
LangGraph or similar agent framework packaged as a service
Local K8s cluster (minikube, kind, or k3d)

Setup steps

Containerize a LangGraph agent with multi-stage Dockerfile
Write Helm chart: Deployment, Service, HPA, ConfigMap, Secret templates
Configure liveness (agent health) and readiness (model loaded) probes
Set HPA on custom metric: agent_queue_depth from Prometheus adapter
Store API keys in K8s Secrets, mount via env vars
Deploy to local cluster and run load test triggering scale-up

Features to build

Containerized agents
Helm charts
HPA autoscaling
Liveness/readiness probes
Secrets management

Expected result

Deploy the agent via Helm, run a load test that triggers HPA scale-up from 2 to 10+ pods, perform a rolling update with zero failed requests, and show Grafana panels for queue depth and pod count.

Resume bullet points

→Deployed production AI agents on Kubernetes with HPA scaling to 50 pods
→Built Helm charts enabling zero-downtime agent rollouts

Interview questions

Why scale agents on queue depth instead of CPU?

LLM agents are I/O-bound waiting on API responses — CPU stays low while queue backs up. Queue depth directly measures pending work and triggers scaling before latency degrades.

How do you achieve zero-downtime deploys for agent pods?

Rolling update with maxUnavailable=0, readiness probe confirming model is loaded before receiving traffic, and preStop hook draining in-flight requests before pod termination.

How do you manage LLM API keys in Kubernetes?

Store in K8s Secrets or External Secrets Operator synced from Vault/AWS SM. Mount as env vars, never in images or ConfigMaps. Rotate via secret update + rolling restart.