Rate Limits

Agents burst many LLM + embedding calls. Hitting rate limits causes cascading failures without backoff and queuing.

Rate limits are highway toll booths — too many cars at once and everyone waits; spread traffic or use express lanes.

Scenario

Batch indexing job hits OpenAI 429s. Add semaphore (10 concurrent), exponential backoff, and resume from checkpoint.

Solution

In Production Agent Engineering, apply Rate Limits to this scenario: Batch indexing job hits OpenAI 429s. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Rate Limits (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Rate Limits happens in the code.

1# Rate Limits — minimal example2from openai import OpenAI3
4client = OpenAI()  # create API client5
6# Ask the model to explain this topic7response = client.chat.completions.create(  # core API call for Rate Limits8    model="gpt-4o-mini",9    messages=[10        {"role": "system", "content": "You explain rate limits clearly."},11        {"role": "user", "content": f"What is rate limits?"},12    ],13    temperature=0,14)15print(response.choices[0].message.content)  # show output for debugging

• Treating Rate Limits as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for rate limits

Quick recap — the most important points from this module.