Computer Use

• Key points are in the visual diagram above.

Many enterprise tools (ERP, legacy desktop apps, design software) have no API and no web UI. Computer use agents can operate these systems the way a human would, unlocking automation for the long tail of GUI-only software.

Computer use is a robot sitting at a desk — it sees the monitor, moves the mouse, types on the keyboard, and clicks icons in any application.

Scenario

Agent opens Excel, imports CSV from Downloads, runs a pivot table, exports PDF, and emails it — all via GUI interaction because the company ERP has no API.

Solution

In Browser & Computer Use Agents, apply Computer Use to this scenario: Agent opens Excel, imports CSV from Downloads, runs a pivot table, exports PDF, and emails it — all via GUI interaction because the company ERP has no API. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Do this before moving to the next module — reading alone is not enough.

Open the Code Walkthrough below and run it locally. Change one parameter related to Computer Use (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Highlighted lines show where Computer Use happens in the code.

1# Computer use loop (conceptual)2while not task_complete:3    screenshot = capture_screen()4    action = agent.decide(screenshot, task, history)5  # action: {"type": "click", "x": 450, "y": 320}6    execute(action)7    history.append(action)

• Treating Computer Use as a black box without evaluation
• Ignoring cost and latency in production
• Skipping error handling for computer use

Quick recap — the most important points from this module.