Cost Optimization

Production agents can burn thousands per day. Engineers who optimize cost without killing UX are highly valued — and this is a frequent system design follow-up.

Airline yield management: same destination, but coach vs business class depending on urgency and budget — not everyone gets the flagship model.

Scenario

Support bot costs $8K/month. You add intent router (80% to mini model), semantic cache for top 200 FAQs, and reranker-only on low-confidence retrieval — 55% cost drop with <2% CSAT dip.

Solution

In Interview & System Design, apply Cost Optimization to this scenario: Support bot costs $8K/month. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Cost Optimization (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Cost Optimization happens in the code.

1# Cost Optimization — minimal example2from openai import OpenAI3
4client = OpenAI()  # create API client5
6# Ask the model to explain this topic7response = client.chat.completions.create(  # core API call for Cost Optimization8    model="gpt-4o-mini",9    messages=[10        {"role": "system", "content": "You explain cost optimization clearly."},11        {"role": "user", "content": f"What is cost optimization?"},12    ],13    temperature=0,14)15print(response.choices[0].message.content)  # show output for debugging

• Treating Cost Optimization as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for cost optimization interview

Quick recap — the most important points from this module.