How Computer Use works

How Computer Use Works with Clod

• Core Mechanism: Computer use is implemented using the exact same underlying tool system as general tool use.
• General Tool Use Flow:
• User sends a request to Clod (User message + Tool schema).
• Clod analyzes the message and decides if a tool is needed.
• Clod responds with a "tool use part" (containing tool ID, name, and required input).
• The server executes the necessary code/action.
• The server sends the result back to Clod via a "tool result part. "
• Computer Use Implementation:
• A special, small tool schema is sent to Clod, which expands into a large structure detailing available actions (e. g. , mouse move, left click, screenshot).
• Clod decides to use an action function and sends a "tool use part. "
• Developer Responsibility: The developer must fulfill the tool use request.
• Simulation: To simulate a computing environment, a Docker container is used to programmatically execute requested key presses and mouse movements.
• The result of the action is then sent back to Clod.
• Key Takeaway: Clod does not directly manipulate a computer; the tool system facilitates the action, and the developer provides the actual computing environment (e. g. , via a Docker container).
• Getting Started:
• Anthropic provides a reference implementation (a Docker container) that handles tool requests and executes programmatic actions.
• Setup requires: Docker installation and a local AWS profile.

Takeaways

• Computer use is implemented using the exact same underlying tool system as general tool use.
• Clod analyzes a request and responds with a "tool use part" detailing the required action, which is then executed by the server.
• A special tool schema expands into a large structure detailing available actions (e. g. , mouse move, left click).
• The developer is responsible for fulfilling the tool use request, typically by using a Docker container to programmatically execute actions.
• Clod does not directly manipulate a computer; the tool system facilitates the action, and the developer provides the actual computing environment.