Direct Screen Manipulation by Frontier Foundation Models
Anthropic has released a public beta capability allowing its flagship Claude 3.5 Sonnet model to interact directly with standard computer operating systems—interpreting screen pixel feeds, moving mouse pointers, clicking buttons, and typing text.
Referred to as “Computer Use,” the feature gives autonomous software agents the ability to navigate legacy enterprise software, web portals, and complex spreadsheets that lack public REST or GraphQL APIs.
“We are teaching models to use computers the way people do: by looking at the monitor and taking physical GUI actions,” explained Anthropic’s developer relations engineering lead.
Rigorous Safety Enclaves & Permissions Guardrails
Because Computer Use grants models access to active desktop sessions, Anthropic has implemented multi-tiered authorization prompts, domain-block lists, and automatic kill-switches to prevent unintended actions across financial or sensitive data silos.