Video Agents

Video contains temporal information text and images can't capture — tutorials, surveillance, sports analysis, user testing recordings. Video agents unlock automation over this rich medium.

A video agent is a sports analyst who watches game footage, identifies key plays, and explains what happened and why.

Scenario

UX researcher uploads 20 user testing sessions → video agent extracts moments where users struggle, timestamps them, categorizes issues, generates highlight reel.

Solution

In Voice & Multimodal Agents, apply Video Agents to this scenario: UX researcher uploads 20 user testing sessions → video agent extracts moments where users struggle, timestamps them, categorizes issues, generates highlight reel. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Video Agents (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Video Agents happens in the code.

1# Video Agents — minimal example2from openai import OpenAI3
4client = OpenAI()  # create API client5
6# Ask the model to explain this topic7response = client.chat.completions.create(  # core API call for Video Agents8    model="gpt-4o-mini",9    messages=[10        {"role": "system", "content": "You explain video agents clearly."},11        {"role": "user", "content": f"What is video agents?"},12    ],13    temperature=0,14)15print(response.choices[0].message.content)  # show output for debugging

• Treating Video Agents as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for video agents

Quick recap — the most important points from this module.