Cost Optimization

Agent costs scale with usage, not seats. A runaway agent loop or unbounded context window can burn thousands of dollars overnight. Production teams need deliberate strategies to control spend while maintaining user experience.

Cost optimization is meal planning for a restaurant — you buy ingredients in bulk, portion carefully, and use cheaper substitutes where diners won't notice the difference.

Scenario

Support agent caches answers for top 200 FAQs (semantic similarity > 0.95) — 40% of queries hit cache at $0. Summarizes conversation every 10 turns instead of sending full history — token usage drops 60%.

Solution

In Production Agent Engineering, apply Cost Optimization to this scenario: Support agent caches answers for top 200 FAQs (semantic similarity > 0. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Cost Optimization (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Cost Optimization happens in the code.

1def route_model(query: str, history_len: int) -> str:  # define a reusable function2    if is_faq(query) and cache_hit(query):3        return "cached"  # $04    if len(query) < 100 and history_len < 3:5        return "gpt-4o-mini"  # cheap6    if requires_reasoning(query):7        return "o1-mini"  # return the result8    return "gpt-4o-mini"  # return the result

• Treating Cost Optimization as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for cost optimization

Quick recap — the most important points from this module.