Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?

Imagine you're building an AI assistant that needs to navigate various applications on a computer, like managing emails, scheduling meetings, or pulling up documents. The challenge isn't just getting the right answer; it's about ensuring the assistant understands the context of its actions and can adapt when things go wrong. Current benchmarks often focus on final outcomes or isolated tasks, which can overlook real-world issues like losing track of context, misinterpreting actions, or failing to recover from errors. This is what's called state verification and context-aware control, and it's crucial for reliable performance in dynamic environments.

To address these shortcomings, the authors developed Desktop-Delta Bench (DDB), a new benchmark designed to evaluate agents on a step-by-step basis across various applications and tasks. DDB includes over 2,000 human-verified instances that test agents on their ability to reconstruct causal transitions and verify their actions in real-time. By focusing on both temporal ordering and action recognition, DDB provides a more nuanced view of an agent's capabilities, revealing gaps in current models. This benchmark not only highlights the importance of context in agent performance but also sets the stage for targeted improvements in how we build and evaluate desktop agents, making them more reliable and effective in real-world scenarios.

Introduces a new benchmark that addresses critical gaps in agent evaluation.

Evaluates multiple model families with a comprehensive dataset, though results show some limitations.