Computer Use Is Evolving Fast

Computer-use systems have crossed an important threshold. They are no longer confined to demonstrations in which a model recognizes a button and clicks it once. Current agents can navigate browsers, manipulate desktop applications, move through multi-step workflows and operate software through the same visible interface that a human uses.

The pace of capability improvement has been substantial. The economics, however, have developed along a more complicated path.

A conventional software integration can execute a structured operation through an API almost instantly. A computer-use agent may instead need to observe a screen, interpret what it sees, determine the next action, execute that action, inspect the result and repeat the cycle many times. For a long workflow, perception, reasoning, interaction and recovery can become a much larger computational workload than the underlying business operation itself.

That makes computer use a particularly interesting systems problem: its value depends not only on what the model can accomplish, but on how much computation, time and infrastructure are required to accomplish it.

1. From Screenshot Control to General Computer Interaction

The modern computer-use paradigm began with multimodal models learning to interpret graphical interfaces and convert natural-language instructions into mouse and keyboard operations.

Anthropic's October 2024 release exposed computer use through an API that allowed Claude to perform actions such as opening browsers, moving the cursor, navigating pages and completing forms. On OSWorld, Claude 3.5 Sonnet achieved 14.9% in the screenshot-only setting and 22.0% when given more execution steps. Anthropic also reported that operations that humans perform almost automatically including scrolling, dragging and zooming were still difficult for the model. (anthropic.com)

OpenAI's Computer-Using Agent (CUA), introduced in January 2025, pushed the approach further. CUA combined GPT-4o's visual capabilities with reinforcement learning and was trained to interact with graphical user interfaces through screen perception plus virtual mouse and keyboard controls. In OpenAI's published evaluation, CUA achieved 38.1% on OSWorld, versus 22.0% for the previous state of the art and 72.4% for humans. On WebArena it reached 58.1%, compared with 36.2% for the previous state of the art and 78.2% for human performance. (openai.com)

The subsequent rise of Claude Sonnet 4.5 illustrates how quickly the field has been moving. Anthropic reported 61.4% on OSWorld, up from 42.2% for Claude Sonnet 4 only four months earlier. (anthropic.com)

The improvement is real. What has also become clear is that benchmark results depend heavily on what the benchmark exposes and how closely its environment resembles actual software use.

2. Why Benchmark Scores Need Context

The original OSWorld benchmark contains 369 tasks spanning real web and desktop applications, operating-system file manipulation and workflows that cross multiple applications.

In the original OSWorld paper, humans completed more than 72.36% of the tasks, whereas the best reported model achieved 12.24%. The researchers found major difficulties in GUI grounding and operational knowledge. (arxiv.org)

A separate benchmark, WebGames, was designed to stress the kinds of basic interactions that humans often find almost effortless. It contains 150 interactive web challenges covering visual comprehension, real-time responsiveness, technical interaction, cognitive capabilities and workflow automation.

The published results put the best AI system at about 48% success versus 95.7% for humans. An earlier benchmark version reported 43.1% AI success against the same 95.7% human figure. (openreview.net)

This creates an interesting contrast. A model may demonstrate exceptional performance in reasoning, coding or knowledge tasks and nevertheless struggle with apparently simple actions such as positioning a cursor, reacting to a changing screen or maintaining the correct interface state.

The bottleneck is not necessarily intelligence in the abstract. It can be interaction itself.

3. OSWorld 2.0 Raises the Difficulty

The next-generation OSWorld 2.0 benchmark was designed specifically to expose weaknesses that shorter and more static evaluations can hide.

It contains 108 long-horizon workflows covering everyday and professional activities. The benchmark introduces dynamic interfaces, cross-source reasoning, hidden application state, visual-spatial requirements, information that can arrive during execution and the need for extended recovery over long sequences.

The scale of those workflows is very different from earlier benchmarks. Human users took a median of about 1.6 hours to complete an OSWorld 2.0 task. In the reported evaluation setup, Claude Opus 4.7 using maximum thinking required approximately 318 tool calls per task, compared with about 30 tool calls in OSWorld 1.0. (arxiv)

This distinction is critical for economics. A benchmark involving a few interface actions may tell us whether the model can operate a GUI. A workflow requiring hundreds of interactions tells us much more about whether the technology can operate that GUI efficiently.

4. What Current Frontier Systems Actually Cost

OSWorld 2.0 provides one of the clearest quantitative views of the current trade-off.

Source: OSWorld 2.0 evaluation. (alphaxiv.org)

These figures make the economics visible. In the reported benchmark configuration, the strongest system did not simply consume more compute; it consumed substantially more compute while still achieving only 20.6% binary task completion.

5. Cost Per Attempt Is Not the Same as Cost Per Success

An even more useful metric can be derived from the OSWorld 2.0 data:

expected inference cost per successful completion ≈ cost per attempted task ÷ binary success rate

Using the published values:

These values are derived calculations, rather than additional measurements published by the benchmark authors. They assume independent repeated attempts until success and should therefore be viewed as an expected inference-cost indicator, not a guaranteed production expenditure.

Even with that caveat, the calculation exposes an important property of agent economics: a low per-attempt price can coexist with a high expected cost per completed outcome when the probability of success is low.

For production systems, that distinction is more useful than the token price alone.

6. Putting Human Labor and Agent Compute on the Same Scale

The OSWorld 2.0 benchmark reports a 1.6-hour median human completion time.

That allows a simple break-even calculation.

For a task with that duration:

GPT-5.5: $25.50 ÷ 1.6 hours ≈ $15.94/hour

Claude Opus 4.8: $72.40 ÷ 1.6 hours ≈ $45.25/hour

These numbers should not be interpreted as direct employee-cost comparisons. Human labor includes salary, benefits, management, training, equipment and other overhead, while the benchmark's agent figures represent estimated task-level inference costs under a specific configuration.

The calculation instead establishes an economic threshold.

If the human equivalent really requires 1.6 hours, then an agent costing tens of dollars in inference may be economically attractive.

If the same automation is being used for a task that a person completes in 30 seconds, the economics change dramatically.

The duration and value of the human alternative therefore matter just as much as the model's raw price.

7. The Computer-Use Loop Has a Structural Cost

OpenAI describes CUA as an iterative process in which the model receives visual information, reasons over the current state and produces actions such as clicking, scrolling or typing before observing the result and continuing.

Steelman Labs characterizes the prevailing design in similar terms:

1. capture the current screen and environment representation,
2. run an LLM over that state,
3. issue a tool action,
4. execute it,
5. repeat.

(steelmanlabs.com)

That architecture creates a peculiar cost distribution.

A human looking at a button labeled “Submit” does not perform a large reasoning computation to decide what the button means.

A computer-use agent may instead spend computation identifying the visual element, determining its coordinates, generating the action, executing it, capturing the resulting screen and verifying the changed state.

Consequently, the mouse click is inexpensive.

The machinery surrounding the click is what consumes the compute.

8. Perception and Control Can Consume the Majority of the Work

The OSWorld 2.0 analysis indicates that visual grounding, low-level interaction and tool-use overhead can dominate the agent's action budget. Steelman Labs highlights the same architectural issue: large models can spend a substantial portion of their computation simply perceiving and manipulating the interface rather than performing the higher-level reasoning needed to solve the task. This creates a mismatch between how humans and current agents allocate computation.

Humans have specialized perceptual and motor capabilities that operate extremely quickly.

An LLM-centric agent may repeatedly use a large multimodal model to do jobs as elementary as finding an interface element, moving a cursor or deciding that the current page has finished loading.

In long workflows, this becomes a major efficiency problem.

9. More Steps Mean More Opportunities to Pay

The OSWorld Human study examined exactly how much interaction an agent needs relative to a human trajectory.

The researchers evaluated 16 agents against manually annotated human trajectories. Their original paper found that even leading agents used 1.4×–2.7× more steps than necessary; the later MLSys conference version reports 2.7×-4.3× under its expanded methodology. (arxiv)

The precise range depends on the evaluation setup, but the core result is consistent: agents frequently arrive at the same result through substantially longer action sequences.

That extra interaction is economically relevant because each additional step can create more inference, more visual processing, more context, more latency and another opportunity to make an error.

Human efficiency is therefore not simply about being faster at executing a mouse click. It is also about requiring fewer intermediate actions.

10. Token Consumption Can Become Extremely Large

The model does not spend compute only when it decides to click.

It also spends compute representing and reasoning over the state of the workflow.

OSWorld 2.0 makes this visible. In the reported batched-action configuration, Claude Opus 4.8 generated approximately 224,000 output tokens per task, while GPT-5.5 generated approximately 37,100 output tokens per task. (alphaxiv.org)

That is an enormous difference in token consumption.

It demonstrates why model selection cannot be based purely on task-completion percentage.

Two models may both complete a portion of the workload successfully while consuming radically different quantities of inference.

In high-volume environments, token efficiency can therefore become an economic differentiator in its own right.

11. Public Pricing Makes the Mechanism Visible

OpenAI's current model documentation lists computer-use-preview at $3 per million input tokens and $12 per million output tokens, while also noting that tool-specific models can have additional per-tool-call charges. (developers.openai.com)

OpenAI's broader pricing page currently lists computer-use-preview at $1.50 per million input tokens and $6 per million output tokens under its standard current pricing table. This illustrates an important practical point: production cost analysis has to use the actual commercial pricing applicable to the deployment configuration because provider pricing can change. (platform.openai.com)

The structural principle is nevertheless stable.

If a workflow repeatedly consumes large context, produces long reasoning traces and executes many actions, the total inference bill scales with the trajectory rather than with the apparent simplicity of the individual action.

12. Reliability Is a Different Metric From Capability

A second problem emerges when computer-use systems are tested repeatedly.

The 2026 study On the Reliability of Computer Use Agents investigated repeated executions of the same task and found that an agent could succeed once and then fail when the same task was run again.

The paper attributes this instability to three broad factors:

execution stochasticity, ambiguity in the task specification and variability in agent behavior. (alphaxiv.org)

This matters because conventional benchmark metrics can reward isolated successes.

The reliability paper contrasts standard Pass@k, which captures whether a system succeeds at least once across multiple attempts, with Pass^k, which measures success across repeated executions. Its conclusion is that production systems require evaluation of consistency, not simply evidence that an agent can eventually get a task right. (alphaxiv.org)

For workflows in medicine, finance, compliance, legal operations or other regulated settings, that distinction is fundamental.

A system that occasionally performs a task correctly is not equivalent to one that performs it consistently.

13. Some Agents Avoid the Interface Instead of Using It

OSWorld 2.0 also exposes another limitation in present-day computer use.

Steelman Labs reports examples in which frontier models completed benchmark tasks by bypassing the intended graphical interface. For example, models reportedly used internal APIs to make a hotel reservation or purchase a train ticket, and another task involving GIMP was solved through Python rather than the intended GUI interaction. (steelmanlabs.com)

From a pure optimization perspective, this behavior is understandable.

Why click through twelve web pages when a structured endpoint can perform the same operation directly?

But that behavior highlights where computer use is genuinely valuable.

An API call is generally faster, cheaper and more deterministic than reconstructing the same operation through a visual interface.

The GUI becomes economically important when the GUI is the only practical access mechanism.

14. Choosing Between APIs, Workflow Automation and Computer Use

A practical architecture therefore emerges.

For structured systems with stable interfaces and well-defined inputs and outputs, APIs remain the preferred mechanism.

For business processes that are largely deterministic but span applications with available connectors and triggers, traditional workflow automation is often more appropriate.

Computer-use agents are most compelling when the software offers no convenient API, when multiple unrelated applications must be navigated, when important information exists only within the interface, or when the workflow is too variable to justify building a large collection of brittle automation rules.

In other words, computer use should be evaluated as an integration strategy, not simply as another model capability.

A More Efficient Computer-Use Architecture

This architecture avoids making a large reasoning model the default engine for every low-level interaction. The expected effect is lower token usage, lower latency and potentially better reliability.

15. Where Computer Use Can Actually Make Economic Sense

Consider a process that requires an employee to log into a legacy government portal, move through several pages, retrieve a document, cross-check information, upload another document, respond to an unpredictable validation message and finally submit the application.

A conventional API-first strategy may be impossible because the system does not expose a public API. A deterministic RPA implementation may work until the interface changes. A human can adapt to those changes, but does so at recurring labor cost. A computer-use agent has a different advantage: it can interact directly with the existing interface without requiring a bespoke integration.

This is the category where the technology becomes economically interesting because the agent's flexibility substitutes for integration effort.

16. Compute Has an Environmental Dimension Too

The economics of computer use are connected to infrastructure consumption.

Microsoft Research's 2025 analysis estimated a median 0.34 Wh per query for frontier-scale models above 200 billion parameters under realistic H100-based workloads, with an interquartile range of 0.18–0.67 Wh. (microsoft.com)

Google has separately estimated the median energy associated with a Gemini Apps text prompt at approximately 0.24 Wh, together with approximately 0.03 gCO₂e and 0.26 mL of water, based on Google's own infrastructure and measurement methodology. (cloud.google.com)

These numbers are not direct measurements of a full computer-use workflow and should not be interpreted as such.

The relevant relationship is instead:

energy per task ≈ energy per inference × number of inference events

A workflow that triggers dozens or hundreds of model interactions can therefore have a very different environmental footprint from a single ordinary prompt.

Compute efficiency is thus both a financial issue and an infrastructure issue.

17. Total Automation Cost Extends Beyond Model Inference

The model bill is only one part of the economics. A production system must also account for the consequences of failure and the infrastructure required to operate safely. Retries can cause additional inference. Human handoffs introduce labor costs. Verification adds computation or operational overhead. Browser sessions, virtual machines, containers, networking and storage add infrastructure expenses. Monitoring and execution tracing are required to diagnose failures. Security controls become more substantial when an AI can directly modify external systems. The true production metric is consequently closer to:

model inference + tool calls + infrastructure + retries + human intervention + verification

rather than simply the published token rate.

18. The Metric That Matters Most

For enterprise deployment, “dollars per million tokens” is no longer a sufficient economic metric.

A stronger measure is:

Cost per verified successful workflow

That figure should incorporate the full operational path:

inference + tool calls + infrastructure + retries + human intervention + verification

and compare it with the cost of achieving the same verified outcome through a person, API integration, RPA system or another automation mechanism. This changes the evaluation question from model-centric economics to outcome-centric economics.

It also prevents a common analytical mistake: assuming that a lower token price automatically means a cheaper automation platform.

19. The Capability Curve Is Clearly Moving Upward

The progress is significant.

The original OSWorld results reported approximately 12.24% model success. Anthropic subsequently reported 22.0% for Claude 3.5 Sonnet with additional execution steps. OpenAI's CUA reached 38.1% on its published OSWorld comparison, and later systems produced substantially higher scores on evolving versions of the benchmark. (arxiv)

Anthropic's own results show the pace particularly clearly: 42.2% for Claude Sonnet 4 became 61.4% for Claude Sonnet 4.5 over roughly four months.

Yet the shift to OSWorld 2.0 also shows why raw benchmark improvement cannot be viewed in isolation.

Once the tasks become longer, dynamic and representative of professional workflows, the difficulty rises sharply. Under OSWorld 2.0, the best reported binary-completion result is 20.6% for Claude Opus 4.8, alongside approximately $72.40 estimated cost per task, 224K output tokens and 481.8 tool calls in the reported batched configuration. (alphaxiv.org)

Computer use is consequently advancing on two fronts simultaneously:

models are improving, while evaluation is becoming more realistic.

20. Where Computer Use Becomes Economically Attractive

The economics become strongest when several conditions coincide. The automated process should create enough value that reducing human effort matters materially. The underlying software should be difficult or impossible to integrate through conventional APIs. The workflow should contain enough complexity or variability that hard-coded automation becomes expensive to maintain. Failures should be detectable and recoverable without causing unacceptable business consequences. And the task should be long or expensive enough in human terms to justify the inference and infrastructure required by the agent. When those conditions are absent, simpler automation generally remains more attractive.

The evidence suggests a layered strategy.

Use deterministic code and APIs whenever the underlying software already exposes a clean, reliable interface.

Use workflow automation when the business process is structured but spans systems that can be connected through predefined rules and integrations.

Use LLM reasoning when interpretation, planning or ambiguity genuinely requires model intelligence.

Use computer use when the graphical interface is effectively the only available integration surface.This is not a statement that computer use is inferior to other forms of automation.It is a statement about where its unique flexibility has economic value.

Conclusion

Computer-use agents are improving rapidly, but capability alone does not determine production readiness.

Their effectiveness depends on reliability, latency, compute consumption, and cost per successful workflow. For stable, deterministic processes, APIs and workflow automation remain more efficient; computer use becomes valuable when software is difficult to integrate, highly variable, or accessible primarily through a GUI.

The next stage of computer-use systems is therefore not simply larger models, but more efficient architectures that allocate expensive reasoning only where it is needed.

The real benchmark is simple: verified outcomes per dollar, second, and unit of compute.