Production Debugging

Agents fail opaquely: wrong answer could be bad retrieval, tool timeout, or prompt drift. Interviewers want a systematic debugging playbook.

Flight recorder analysis: reconstruct the whole flight path, not just 'we crashed near the lake.'

Scenario

Users report wrong refund policy answers after deploy. Trace shows retrieval returning archived 2023 doc — root cause: missing `effective_date` filter in new indexer config.

Solution

In Interview & System Design, apply Production Debugging to this scenario: Users report wrong refund policy answers after deploy. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Production Debugging (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Production Debugging happens in the code.

1# Production Debugging — minimal example2from openai import OpenAI3
4client = OpenAI()  # create API client5
6# Ask the model to explain this topic7response = client.chat.completions.create(  # core API call for Production Debugging8    model="gpt-4o-mini",9    messages=[10        {"role": "system", "content": "You explain production debugging clearly."},11        {"role": "user", "content": f"What is production debugging?"},12    ],13    temperature=0,14)15print(response.choices[0].message.content)  # show output for debugging

• Treating Production Debugging as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for production debugging

Quick recap — the most important points from this module.