Screen Understanding

Computer use and browser agents need to 'see' the UI before clicking. Screen understanding bridges raw pixels (or DOM) and agent reasoning, grounding actions in what's actually visible.

Screen understanding is reading a restaurant menu before ordering — you look at what's available, then decide what to ask for.

Scenario

Agent receives screenshot of settings page → identifies 'Privacy' section at coordinates (320, 450) → clicks it → reads new screen → toggles 'Share analytics' off.

Solution

In Voice & Multimodal Agents, apply Screen Understanding to this scenario: Agent receives screenshot of settings page → identifies 'Privacy' section at coordinates (320, 450) → clicks it → reads new screen → toggles 'Share analytics' off. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Screen Understanding (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Screen Understanding happens in the code.

1# Set-of-Mark approach2annotated = overlay_element_numbers(screenshot, accessibility_tree)  # key line for Screen Understanding3response = vision_llm.invoke([4    {"type": "image_url", "image_url": {"url": annotated_url}},5    {"type": "text", "text": "Click the element to disable analytics. Return element number."},6])7element_id = parse_element_number(response)8click(accessibility_tree[element_id].bounds)

• Treating Screen Understanding as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for screen understanding

Quick recap — the most important points from this module.