How computer use works

Tool Use and Computer Use Concepts

• Tool Use Mechanism:
• To use a tool with Claude, a request is sent containing a user message and a tool schema.
• The tool schema describes the additional functionality available (e. g. , a "get weather" function).
• Claude analyzes the user message and decides if a tool is needed.
• If a tool is used, Claude responds with a "tool use" part, including a tool use ID, name, and required input (e. g. , location).
• The external server code receives the tool use request, executes the necessary action (e. g. , fetching weather), and returns the result.
• The result is sent back to Claude in a "tool result" part for summarization.

• Computer Use Implementation:
• Computer use follows the exact same tool use flow.
• A special, small tool schema is sent to Claude, which is internally expanded into a much larger structure.
• This large structure informs Claude that it can call an action function with arguments like "mouse move," "left click," and "screenshot. "
• When Claude decides to use a tool, the developer must fulfill the request.
• To simulate a computing environment, a program (like a Docker container) is used to programmatically execute key presses and mouse movements based on Claude's request.
• The final result of the action is sent back to Claude.
• Key Takeaway: Claude does not directly manipulate a computer; computer use is implemented by using the tool system, and the developer provides the actual computing environment.

• Getting Started:
• Anthropic provides a reference implementation (a Docker container) that already contains the necessary code to receive tool requests and execute programmatic actions.
• Users can set up this implementation using a simple Docker command to test Claude's computer use functionality.

Takeaways

• Tool use involves Claude analyzing a user message and responding with a structured tool use request (ID, name, input).
• The external server executes the requested action and returns the result to Claude for summarization.
• Computer use is implemented as a specialized tool use flow, allowing Claude to request actions like mouse movement or clicking.
• Claude does not directly manipulate a computer; the developer must provide the execution environment using the tool system.