W&B

Teams already using W&B for ML model training can extend the same workflow to agent prompt iteration, A/B tests, and eval leaderboards.

W&B for agents is a lab notebook that automatically records every experiment and charts which prompt version wins.

Scenario

Log 3 prompt variants on 100 eval questions; Weave leaderboard shows variant B has highest correctness at lowest token cost.

Solution

In Agent Evaluation & Observability, apply W&B to this scenario: Log 3 prompt variants on 100 eval questions; Weave leaderboard shows variant B has highest correctness at lowest token cost. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to W&B (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where W&B happens in the code.

1import weave  # import dependencies2weave.init("support-agent")3
4@weave.op5def answer_question(question: str) -> str:  # define a reusable function6    return agent.invoke(question)  # return the result7
8evaluation = weave.Evaluation(dataset=eval_ds, scorers=[correctness_scorer])9await evaluation.evaluate(answer_question)

• Treating W&B as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for wandb

Quick recap — the most important points from this module.