Scaling

A demo handling 5 users breaks at 500. Agent workloads are spiky (product launches, batch jobs) and expensive (LLM tokens, GPU). Without a scaling strategy, you get timeouts, runaway bills, or both.

Scaling is adding lanes to a highway before rush hour — you expand capacity before traffic jams form, not after cars are stuck for miles.

Scenario

Router classifies queries: 70% go to gpt-4o-mini (fast/cheap), 25% to gpt-4o, 5% to o1 for reasoning. API autoscales 3→15 pods at 1000 RPM. Workers scale 5→50 on queue depth > 100.

Solution

In Production Agent Engineering, apply Scaling to this scenario: Router classifies queries: 70% go to gpt-4o-mini (fast/cheap), 25% to gpt-4o, 5% to o1 for reasoning. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Scaling (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Scaling happens in the code.

1# Scaling — minimal example2from openai import OpenAI3
4client = OpenAI()  # create API client5
6# Ask the model to explain this topic7response = client.chat.completions.create(  # core API call for Scaling8    model="gpt-4o-mini",9    messages=[10        {"role": "system", "content": "You explain scaling clearly."},11        {"role": "user", "content": f"What is scaling?"},12    ],13    temperature=0,14)15print(response.choices[0].message.content)  # show output for debugging

• Treating Scaling as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for scaling

Quick recap — the most important points from this module.