Testing

AI apps are complex and non-deterministic. Without tests, every deploy is a gamble.

Testing is a safety net — unit tests catch code bugs, evals catch quality drops.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

Overview

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Testing catches bugs before users do — unit tests for logic, integration tests for APIs, evals for LLM quality.
2.Unit tests are fast and mock the LLM.
3.Integration tests check API endpoints.
4.Evals measure output quality on golden datasets.
5.Never call real LLM APIs in CI unit tests.

Real Example

Scenario

You changed the chunking logic and need to verify nothing broke before deploying.

What you would do

Run unit tests on chunking (deterministic), integration test on POST /chat (mocked LLM), and check eval scores against golden Q&A pairs.

Practice Task

Create test_math.py with a function add(a, b) and a pytest test_add() that asserts add(2, 3) == 5. Run pytest -v. Then add a test that mocks an API call using unittest.mock.patch.

Commands

Commands to Remember

pytest # run all tests in the project
pytest tests/test_file.py -v # run one test file with verbose output
pytest -k "test_name" # run only tests matching a name
pytest --cov=src # run tests and show code coverage

Cheat Sheet

Quick recap

quick ref

•Unit tests — fast, isolated, mock the LLM
•Integration tests — API endpoints with test database
•Evals — measure LLM output quality on golden datasets
•Never call real LLM APIs in CI unit tests
•Test retrieval and chunking logic separately from generation
•pytest + unittest.mock.patch for mocking

Common Mistakes

✕Real LLM APIs in unit tests
✕Testing LLM output verbatim
✕No tests for retrieval/chunking logic