Claude SDK Computer Use Tool Tutorial: Build a Desktop Agent with the Toolset Pattern

Introduction

Computer use gives Claude a mouse, keyboard, and eyes: your agent looks at screenshots, clicks buttons, types into forms, and drives software that was never built for automation. Since the August 2026 general availability, the mechanism changed shape: instead of declaring a single dated tool, you declare one computer_toolset_20260801 entry and the API expands it into a family of 17 member tools like screenshot, left_click, type, and zoom. The Python SDK goes one step further and ships an abstract class you subclass to connect those members to your own desktop backend, while the SDK runs the tool loop, approval callbacks, and URL and file policies for you. This tutorial is for developers building a Claude-powered desktop agent: you will write the driver subclass, wire a human approval callback, run the agent loop, and migrate an older computer_20251124 integration so it stops failing on Claude 5.5 models.

Quick answer

Subclass the Python SDK’s BetaAbstractComputerToolset20260801, implement screenshot, click, and keyboard members against your own desktop backend, add a confirm callback for approvals, run the beta.messages.tool_runner loop, and declare the toolset as {"type": "computer_toolset_20260801"} with no beta header.

That answer covers the happy path. The rest of this tutorial turns each phrase into working code, with the migration traps and safety rules that bite first.

Evidence

Anthropic’s computer use tool documentation states that one computer_toolset_20260801 entry gives Claude 17 member tools, and that calls arrive as tool_use blocks whose name is the member and which carry "toolset_name": "computer", often several per turn in a batch action. Two numbers matter for budgeting. First, the same docs warn that a long agent loop accumulates screenshots quickly, roughly 1,000 to 1,800 input tokens each, and that once a single request carries more than 20 images every image is held to a stricter per-side limit. Second, Anthropic’s GA announcement (published 2026-08-20) made computer use eligible for HIPAA-regulated workloads under Anthropic’s BAA and notes that the updated tool “lets Claude take several actions per turn instead of one per model call, so tasks finish in fewer calls and less time.” The platform rule that drives the migration section below is documented there too: on the Claude API and Google Cloud, Claude 5.5 and later models accept only the toolset, while Amazon Bedrock keeps accepting computer_20251124 for Opus 5.5, Sonnet 5.5, and Haiku 5.5.

Prerequisites

• A Claude API key with computer use access, set as ANTHROPIC_API_KEY
• Python 3.10 or newer, with the Anthropic Python SDK installed (pip install --upgrade anthropic)
• A disposable Linux desktop the model can see: a Docker container or VM running Xvfb with a lightweight window manager is the recommended setup; Anthropic’s docs list Xvfb, Mutter, and Tint2 as the reference combination
• Comfort reading Python tracebacks, because a computer use integration surfaces backend bugs as model-visible ToolError text

One honest note before you start: the SDK ships the toolset abstraction, not a desktop driver. The driver code below is the part you write against your own backend, and it follows the structure of Anthropic’s own guide in computer-toolset.md, which is the primary source for every method name and callback contract in this tutorial.

Step-by-step guide

Step 1: Verify the SDK ships the toolset classes

The toolset support lives in the SDK’s beta namespace. Confirm the classes exist before writing anything:

pip install --upgrade anthropic
python -c "from anthropic.tools.computer import BetaAbstractComputerToolset20260801; print('computer toolset OK')"
python -c "from anthropic.tools.browser import BetaAbstractBrowserToolset20260801; print('browser toolset OK')"

If the first import fails with ImportError, your SDK is too old; the toolset classes shipped after computer use left beta. This two-line check saves you from writing a driver against an API surface that does not exist.

Step 2: Declare the toolset in a bare API call

Before building the driver, see the wire format. The toolset entry takes no name, no display dimensions, and no beta header:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    tools=[
        # One entry. No name, no display_width_px, no beta header.
        {"type": "computer_toolset_20260801"},
    ],
    messages=[
        {"role": "user", "content": "Take a screenshot and describe what you see."},
    ],
)

Claude’s calls come back as tool_use blocks like {"type": "tool_use", "id": "toolu_...", "name": "screenshot", "toolset_name": "computer", "input": {}}. Notice what moved: in the old computer_20251124 tool the action sat in input.action; in the toolset the action is the block’s name, and toolset_name tells you it belongs to the computer family. Every tool_result you return must echo "toolset_name": "computer", or the API rejects it.

Step 3: Subclass the abstract toolset and implement a driver

This is the heart of the SDK pattern. You subclass BetaAbstractComputerToolset20260801 and implement one method per member tool your backend supports. A member you do not override is advertised to the API as enabled: false, so start with the four members every first agent needs:

from anthropic.tools.computer import (
    BetaAbstractComputerToolset20260801,
    BetaScreenshotResult,
    BetaToolsetCallContext,
)
from anthropic.types.beta import (
    BetaComputerLeftClickInput,
    BetaComputerScreenshotInput,
    BetaComputerTypeInput,
)

class MyDesktop(BetaAbstractComputerToolset20260801):
    def __init__(self, backend, **options):
        super().__init__(**options)
        self.backend = backend  # VNC client, remote-desktop API, or xdotool

    def screenshot(self, context: BetaToolsetCallContext,
                   input: BetaComputerScreenshotInput) -> BetaScreenshotResult:
        return BetaScreenshotResult(data=self.backend.png_base64())

    def left_click(self, context: BetaToolsetCallContext,
                   input: BetaComputerLeftClickInput) -> None:
        self.backend.click(input.coordinate, input.text)

    def type(self, context: BetaToolsetCallContext,
             input: BetaComputerTypeInput) -> None:
        self.backend.type(input.text)

    def close(self) -> None:
        super().close()  # first: it waits for calls in flight
        self.backend.close()

The signatures are contracts, not suggestions: input is the parsed BetaComputer<Member>Input, and the member docstrings in the SDK are the authoritative description of each parameter. input.text on a click is not typed text; it holds modifier keys such as ctrl+shift to hold during the click. One subtle contract from the docs: screenshot must capture at device scale factor 1 in a fixed size for the whole session, because the model expresses every coordinate in the pixel space of the screenshots you return. If you downscale screenshots, scale Claude’s coordinates back up before applying them, or every click lands offset.

When a member raises ToolError, the model reads its text unredacted and the run continues, so keep error strings to fixed phrases. Never interpolate the exception’s own message: an exception from your backend can carry screen content, host names, or session tokens, and those would leak into the model’s context.

Step 4: Add the approval callback

The confirm option is called before every member call runs, and only a return value of True approves. Anything else, or an exception, refuses the call. The SDK requires a confirm callback whenever type, key, or hold_key is enabled, because those members act on whatever has keyboard focus:

import json
from anthropic.tools.computer import BetaComputerConfirmContext

def confirm(context: BetaComputerConfirmContext) -> bool:
    if context.member not in {"type", "key", "hold_key"}:
        return True
    # json.dumps escapes invisible characters; what the approver reads
    # must be exactly what runs, in case the model is relaying screen content
    return input(f"Allow {context.member}? {json.dumps(context.input.to_dict())} [y/N] ") == "y"

desktop = MyDesktop(backend, confirm=confirm)

For production you would replace input() with your own approval surface, but the rule stays the same: show the approver the member, the target, and the input with every invisible character escaped. Anthropic’s docs specifically call out clicks, type, key, and form_input on sites where purchases, messages, or accepting terms can happen as the moments to ask a human.

Step 5: Run the agent loop with the tool runner

The SDK’s beta.messages.tool_runner handles the loop, batch actions, and the halt semantics. Without it you would reimplement the rule that a turn’s actions run in order and stop at the first failure, with unrun actions answered is_error: true and the exact text Not executed: an earlier computer action in this turn failed.

from anthropic import Anthropic

with MyDesktop(backend, confirm=confirm) as desktop:
    runner = Anthropic().beta.messages.tool_runner(
        model="claude-sonnet-5-5",
        max_tokens=16000,
        tools=[desktop],
        messages=[
            {"role": "user",
             "content": "Open the file manager, create a folder named invoices, and confirm it exists."},
        ],
    )
    for message in runner:
        print(message)

You close the desktop; the runner never does, so one instance can serve several runs. If you write the loop by hand instead, remember three rules from the docs: iterate over every tool_use block in the turn rather than reading only the first, dispatch on name together with toolset_name, and echo "toolset_name": "computer" on each tool_result, because a result without it is rejected with an invalid_request_error.

Step 6: Migrate an existing computer_20251124 integration

If you have a working integration on the older dated tool, Anthropic publishes a nine-step upgrade checklist in the computer use tool docs. The moves that catch teams:

1. Remove the beta header. Drop anthropic-beta: computer-use-2025-11-24 from requests; in the SDK, remove the betas parameter and call the Messages API through the standard client rather than the beta namespace.
2. Change the tools entry. Set type to computer_toolset_20260801 and delete name, display_width_px, display_height_px, display_number, and enable_zoom. The toolset rejects each of these fields outright.
3. Decide about zoom. Zoom is enabled by default on the toolset, while enable_zoom defaulted to false. If your environment cannot produce zoom images, withhold it with "configs": {"zoom": {"enabled": False}} rather than leaving it enabled and returning errors.
4. Handle every block in a turn. Iterate over all tool_use blocks, dispatch on name plus toolset_name instead of input.action, which no longer exists in member inputs.
5. Run blocks in order and use the halt text. Stop the batch at the first failure and answer later blocks with Not executed: an earlier computer action in this turn failed.
6. Echo toolset_name on results. Every member tool_result carries "toolset_name": "computer", and results may contain only text and image content.
7. Support repeat on key. The key member accepts an optional repeat count from 1 to 100; a handler that ignores the field presses the key once.
8. Resize screenshots yourself. The toolset rejects images over the model’s image limits instead of downscaling them, and your coordinates live in screenshot pixels, so keep the scale factor to map them back.
9. Remove unsupported options. Move any defer_loading from the entry into configs.

The two forms cannot share a request: the entry cannot be declared alongside a computer_20251124 entry or another tool named computer. On the Claude API and Google Cloud, send the old tool to a Claude 5.5 model and you get an HTTP 400; plan the migration before you move any production workload to Opus 5.5, Sonnet 5.5, or Haiku 5.5. Amazon Bedrock is the exception and still accepts the older tool for the 5.5 models.

Step 7: Keep the screenshot budget under control

Two documented numbers set your context budget. Each screenshot costs roughly 1,000 to 1,800 input tokens, and a request carrying more than 20 images gets stricter per-side limits on every image. Anthropic’s sizing guidance: for general desktop tasks use 1024x768 or 1280x720, avoid resolutions above 1920x1080, encode as base64 PNG or JPEG, and prune old screenshots in batches, not one per turn, so prompt caching prefixes stay byte-identical between prune events. One platform-specific trap: on Claude Opus 5.5, Sonnet 5.5, and Haiku 5.5, do not prune on the client, because removing an earlier screenshot invalidates every later thinking block. Resize screenshots to 2000 px or less per side instead and use server-side tool result clearing.

Troubleshooting

HTTP 400 when you only changed the model name to a 5.5 model. Your request still declares the old computer_20251124 tool or still sends the computer-use-2025-11-24 beta header. On the Claude API and Google Cloud, Claude 5.5 and later models accept only the toolset. Apply the migration in Step 6: remove the beta header, set type to computer_toolset_20260801, and delete name and the display-dimension fields. If you run on Amazon Bedrock, the older tool still works there for the 5.5 models, so this error on Bedrock points elsewhere.

invalid_request_error on your tool_result blocks. You forgot "toolset_name": "computer" on a member result, or the result carries content other than text or image. Also check that every tool_use block in a batch got an answer; a request that leaves any block unanswered is rejected. The SDK’s tool_result() helper sets the fields for you.

Clicks land consistently offset in one direction. Coordinates are in screenshot pixels, and you applied them to a display of a different size without scaling. Scale each coordinate by the ratio of your screen size to your screenshot size before clicking, and on macOS Retina displays account for the 2x device pixel ratio. Keeping the capture size fixed for the whole session avoids this class of bug entirely.

Screenshots rejected with a validation error. The toolset does not downscale for you; the image must already fit the model’s image limits. Resize before returning it and keep the scale factor so you can map Claude’s coordinates back to the real screen. The docs suggest 1280x720 as the baseline for desktop tasks.

confirm refuses everything, or the constructor raises about a missing callback. The SDK requires a confirm callback while type, key, or hold_key is enabled. If you only gate keyboard members, return True for the rest as in Step 4; if the desktop runs applications that can send, pay, or delete, gate clicks too. Remember confirm sees the member and its input, not what a click will actually do on screen.

Context exploding after a few dozen turns. You are keeping every screenshot. Resize each so neither side exceeds 2000 px, prune in batches every 25 turns or so keeping the last three screenshots, and put one cache_control breakpoint after the tool definitions and up to three more on the last tool_result of recent turns. On the 5.5 models, resize rather than prune, per the thinking-block invalidation rule above.

FAQ

The FAQ block is in the page front matter, rendered by the site theme.

Conclusion

The toolset pattern splits computer use into two clean halves: Anthropic’s model intelligence and the SDK’s loop, policies, and approvals on one side, and your desktop backend on the other. Subclass the abstract toolset, implement the members your backend supports, gate consequential actions with confirm, and declare the whole thing as one computer_toolset_20260801 entry with no beta header. Migrate off the dated computer_20251124 tool before moving workloads to Claude 5.5 models, and budget your screenshot tokens from the start.

For related ToolPilot coverage, see the Claude dynamic workflows tutorial, which shows how to orchestrate multi-step agent patterns that pair well with a computer use driver; the Claude Haiku 5.5 migration guide, which covers the 5.5 toolset-only behavior across Anthropic, Google Cloud, and Bedrock; and GitHub Copilot computer use in the CLI and desktop app, a vendor-built alternative for teams that would rather not build their own driver.

Frequently asked questions