Trajectory Evaluation

• Key points are in the visual diagram above.

Two agents may reach the same answer via different paths; one may have hallucinated intermediate steps or taken 10x more API calls. Trajectory eval catches inefficient or unsafe paths.

Trajectory eval reviews game replay move-by-move — winning isn't enough if you sacrificed all your pieces to do it.

Scenario

Gold trajectory: search → read_doc → answer. Agent does search → search → search (redundant) → answer. Trajectory eval flags inefficiency despite correct final answer.

Solution

In Eval Engineering & Observability, apply Trajectory Evaluation to this scenario: Gold trajectory: search → read_doc → answer. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Trajectory Evaluation (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Trajectory Evaluation happens in the code.

1def step_efficiency(trajectory, gold_min_steps):  # define a reusable function2    tool_steps = [s for s in trajectory if s["type"] == "tool"]  # key line for Trajectory Evaluation3    return len(tool_steps) <= gold_min_steps * 1.5  # return the result

• Treating Trajectory Evaluation as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for trajectory evaluation

Quick recap — the most important points from this module.