Computer Use Field Notes
OSWorld experiments · July 2026 · every trace public

Eval runs to improve Computer use UX

Three improvement strategies in the report.

01 / SCREEN RESOLUTION
Smaller screenshots lowered cost without lowering pass rate.

Every computer-use step sends a screenshot to the model. Sending 960×540 screenshots used fewer tokens than 1280×720, while the 70-task pass count stayed at least as good.

$7.35per 70 tasks at 960×540
29/70passes versus 25/70 baseline
31.3%fewer model input tokens
280runs across four screenshot sizes on the same 70-task pack
31.3%fewer model input tokens at 960×540 than 1280×720
on pareval score stayed on par despite image compression
1280×720 baseline · $10.10 · 25/70
1024×576 $7.69 · 29/70
960×540 adopted · $7.35 · 29/70
800×450 $6.44 · 24/70
screenshot size resultslower cost · similar pass rate

What we chose

960×540 was the best default in this run: it cost $7.35 per 70 tasks versus $10.10 at 1280×720, and passed 29 tasks versus 25. 800×450 was cheaper, but it passed fewer tasks, so it is a cost-only option rather than the default.

Artifacts for this run

Use these to inspect the resize experiment directly: the run bundle has the trajectories, screenshots, recordings, and score data; the script shows how the 70-task pack was drawn.

Resize artifact bundle S3 · 2.1 GB · 280 trajectories Task-pack script seed 20260704 · reproducible task draw
Control
Keep everything, no task note.
Harness to model Send full history plus the latest screenshot on every step.
Model Re-derive what is done and what remains from many prior screenshots.
Failure mode Long tasks often re-check and retry until they hit the step cap.
29/64 long runs hit the cap · 0.79 finish precision
v1 FocusChain
Append-only prose note, echoed at the end of context.
New input The latest task note is placed after history and screenshot, so it is the last thing the model sees.
New output Each tool call includes a short note: goal, facts, completed work, and remaining work.
Tradeoff The agent stops looping, but the note can self-certify work before the screen proves it.
12.6% fewer steps · 14/64 cap-outs · 0.48 finish precision
v2 Gated Ledger
Structured ledger plus a harness-enforced finish gate.
New schema The note becomes fields for constraints, remaining work, next check, and verified done.
New gate If remaining work is listed, a no-action finish is rejected once and the agent must re-check.
Result The gate worked mechanically, but the model still misread screens and emptied the ledger too early.
10.2% fewer steps · 0.52 finish precision · target missed
long-task outcomesfewer loops · more wrong finishes

What we learned

v1 worked as a termination mechanism: it reduced repeated checking and step-cap failures. v2 proved the next bottleneck was visual verification, not note format. If the model misreads the screen, a structured ledger can still sincerely mark the wrong thing as done.

Artifacts for this run

Use the run bundles to inspect trajectories and screenshots. Use the code links to review exactly how v1 FocusChain and v2 Gated Ledger changed the harness.

Run bundles
Control blocks S3 · 1.0 GB · 80 baseline task-runs v1 FocusChain blocks S3 · 0.97 GB · 80 task-note task-runs v2 Gated Ledger blocks S3 · 1.0 GB · 80 gated-ledger task-runs Long-task extension S3 · 0.72 GB · 46 long-horizon task-runs Smoke runs S3 · 62 MB · v1 and v2 validation probes Analysis and logs S3 · 6.8 MB · scripts, CSVs, task packs, logs
Code and patches
Full source tree S3 · 10 MB · final focuschain-ab branch Algorithm patch series S3 · 21 KB · v1 b66b99dd + v2 2d8ca929 v1 main.py S3 · note capture and echo path v1 context policy S3 · FocusChain context policy v2 main.py S3 · gated ledger and finish gate v2 context policy S3 · structured ledger context policy
2.3×more GUI actions per model call
+3.4%cost movement, not statistically significant
57.5% / 55%batched versus single-action score

Algorithm change tested

The comparison keeps the same model, task pack, prompt cache, screenshot size, and step cap. The change is the tool contract: one arm sends one GUI action at a time, while the batched arm can send several GUI actions from the same model response.

01 Start with the single-action arm: one GUI action per model response.
02 Turn batching on: allow a response to contain multiple GUI actions.
03 Keep the model, screenshots, cache, task pack, and step cap fixed.
04 Run the same 40-task pack and compare paired cost, calls, actions, and score.
Single-action arm The comparison arm sends one click, keystroke, wait, or scroll at a time.
Batched arm The tested arm can send several GUI actions in one model response.
Same setup Both arms use Sonnet, caching, 960×540 screenshots, 50 steps, and the same task pack.
Paired eval Each arm runs the same 40 OSWorld tasks, then bootstrap CIs compare the per-task differences.
Result Batching executed 2.3× more actions per call, but did not produce a clear cost or score win.

What we learned

Batching worked mechanically: it raised GUI actions per model call by 2.3×. But the cost did not clearly fall; the paired estimate moved only +3.4%, with the confidence interval crossing zero. Score stayed on par: 57.5% for batched versus 55% for single-action.

Where to spend the next run

If the goal is cost, batching is not the highest-leverage knob. The better next eval is targeted verification: find traces where batched actions spend many steps without fresh visual feedback, then test whether smaller action groups or screen checks improve reliability.

Artifacts for this run

Only the essentials: raw traces for both arms, the paired summary table, and the patch that makes the comparison explicit. The full bucket still has logs, scripts, and extra code files if needed.

Single-action traces S3 · 632.9 MB · comparison traces Batched traces S3 · 1.0 GB · batching traces Tool-contract patch S3 · exact comparison change Paired summary S3 · calls, cost, score CIs