Release audit

OSWorld V2: 0808 vs 0624

An interactive task-level view of evaluator fixes, anti-cheat mitigations, setup changes, and remaining gaps across 45 changed task definitions.

GitHub issue #34 Markdown report
45changed task definitions
22anti-cheat or integrity changes
17correctness and robustness fixes
6setup, scope, or wording changes

Confirmed cheating fixes

Tasks 055 and 056 were checked against recorded 0624 trajectories that copied the clean reference video and submitted fake or wrapper project files.

TASK 055Exploit blocked

0624 weakness

The clean reference was readable on the Desktop. The agent copied it directly to OSWorld.mp4 and generated a one-clip wrapper MLT.

0808 mitigation

Protected observation media, project re-rendering, export/replay comparison, and perceptual project-resource provenance checks.

Direct validator test: suspicious=true, reason=reference_resource_detected, resource_ssim=1.0
TASK 056Exploit blocked

0624 weakness

The clean 1080p reference was copied as the final MP4. The MLT listed raw assets in its bin but placed only the copied video on the timeline.

0808 mitigation

Only a 480x270 preview reaches the VM. The clean reference stays evaluator-side, and the MLT must contain the real clips, titles, filters, and BGM.

Copied preview: 0/18 sampled frames reached SSIM 0.96; old fake MLT: 0 structural points

Explore every change

Filter by change type or search for a task number, application, weakness, mitigation, or evaluator concept.

No task changes match this filter.

Remaining gaps

Project/export binding: task 056 still does not render the MLT and compare it with the submitted MP4.
Incomplete provenance coverage: task 055 may miss nested, relative, or unusual MLT resources.
Instruction-only controls: some UI-only rules are not technically enforced by evaluators.
Exact-reference shortcuts: tasks 084 and 085 award full component credit for byte-identical hidden-gold artifacts.
Replay tradeoff: task 064 improves workspace integrity but drops fresh evaluator-triggered seeded reruns.
Heuristic rendering: raster thresholds and neutral renderer uncertainty can still create false passes or failures.