Regression Testing

A one-line prompt tweak can break 15% of cases. Automated regression tests make agent development as safe as traditional software engineering.

Agent regression tests are unit tests for behavior — green means ship, red means you broke something you promised worked.

Scenario

PR changes router prompt; CI runs 80-case eval; 6 previously passing billing cases now route to wrong agent — merge blocked.

Solution

In Agent Evaluation & Observability, apply Regression Testing to this scenario: PR changes router prompt; CI runs 80-case eval; 6 previously passing billing cases now route to wrong agent — merge blocked. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Regression Testing (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Regression Testing happens in the code.

1def test_support_regression():  # define a reusable function2    failures = run_eval_suite(agent, dataset="golden_v12", min_pass_rate=0.92)3    assert not failures, f"Regressions: {failures[:5]}"  # key line for Regression Testing

• Treating Regression Testing as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for regression testing

Quick recap — the most important points from this module.