Computer Use Field Notes
OSWorld experiments · July 2026 · three harness experiments

Eval runs to improve Computer use UX

Three improvement strategies in the report.

01 / SCREEN RESOLUTION
Smaller screenshots lowered cost without lowering pass rate.

Every computer-use step sends a screenshot to the model. Sending 960×540 screenshots used fewer tokens than 1280×720, while the 70-task pass count stayed at least as good.

$7.35per 70 tasks at 960×540
29/70passes versus 25/70 baseline
31.3%fewer model input tokens
280runs across four screenshot sizes on the same 70-task pack
31.3%fewer model input tokens at 960×540 than 1280×720
on pareval score stayed on par despite image compression
1280×720 baseline · $10.10 · 25/70
1024×576 $7.69 · 29/70
960×540 adopted · $7.35 · 29/70
800×450 $6.44 · 24/70
screenshot size resultslower cost · similar pass rate

What we chose

960×540 was the best default in this run: it cost $7.35 per 70 tasks versus $10.10 at 1280×720, and passed 29 tasks versus 25. 800×450 was cheaper, but it passed fewer tasks, so it is a cost-only option rather than the default.

Control
Keep everything, no task note.
Harness to model Send full history plus the latest screenshot on every step.
Model Re-derive what is done and what remains from many prior screenshots.
Failure mode Long tasks often re-check and retry until they hit the step cap.
29/64 long runs hit the cap · 0.79 finish precision
v1 FocusChain
Append-only prose note, echoed at the end of context.
New input The latest task note is placed after history and screenshot, so it is the last thing the model sees.
New output Each tool call includes a short note: goal, facts, completed work, and remaining work.
Tradeoff The agent stops looping, but the note can self-certify work before the screen proves it.
12.6% fewer steps · 14/64 cap-outs · 0.48 finish precision
v2 Gated Ledger
Structured ledger plus a harness-enforced finish gate.
New schema The note becomes fields for constraints, remaining work, next check, and verified done.
New gate If remaining work is listed, a no-action finish is rejected once and the agent must re-check.
Result The gate worked mechanically, but the model still misread screens and emptied the ledger too early.
10.2% fewer steps · 0.52 finish precision · target missed
long-task outcomesfewer loops · more wrong finishes

What we learned

v1 worked as a termination mechanism: it reduced repeated checking and step-cap failures. v2 proved the next bottleneck was visual verification, not note format. If the model misreads the screen, a structured ledger can still sincerely mark the wrong thing as done.

2.3×more GUI actions per model call
+3.4%cost movement, not statistically significant
57.5% / 55%batched versus single-action score

Algorithm change tested

The comparison keeps the same model, task pack, prompt cache, screenshot size, and step cap. The change is the tool contract: one arm sends one GUI action at a time, while the batched arm can send several GUI actions from the same model response.

01 Start with the single-action arm: one GUI action per model response.
02 Turn batching on: allow a response to contain multiple GUI actions.
03 Keep the model, screenshots, cache, task pack, and step cap fixed.
04 Run the same 40-task pack and compare paired cost, calls, actions, and score.
Single-action arm The comparison arm sends one click, keystroke, wait, or scroll at a time.
Batched arm The tested arm can send several GUI actions in one model response.
Same setup Both arms use Sonnet, caching, 960×540 screenshots, 50 steps, and the same task pack.
Paired eval Each arm runs the same 40 OSWorld tasks, then bootstrap CIs compare the per-task differences.
Result Batching executed 2.3× more actions per call, but did not produce a clear cost or score win.

What we learned

Batching worked mechanically: it raised GUI actions per model call by 2.3×. But the cost did not clearly fall; the paired estimate moved only +3.4%, with the confidence interval crossing zero. Score stayed on par: 57.5% for batched versus 55% for single-action.

Where to spend the next run

If the goal is cost, batching is not the highest-leverage knob. The better next eval is targeted verification: find traces where batched actions spend many steps without fresh visual feedback, then test whether smaller action groups or screen checks improve reliability.