OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong
Read on arXiv →
Key claim
VLM judges are often unreliable for CUA evaluation.
In plain English
Imagine you're developing an AI that needs to follow complex instructions and adapt its actions based on real-time feedback. The challenge lies in ensuring that the AI can accurately assess its own performance and learn from mistakes, especially when human oversight isn't feasible at scale. Current methods often rely on human annotators or simplistic benchmarks that fail to capture the nuances of real-world tasks, leading to issues like mislabeling failures as successes — a problem known as leniency bias. This can result in agents that seem competent in tests but struggle in practical applications. To address this, the authors created OSReward, a comprehensive benchmark designed to evaluate vision-language models (VLMs) as judges of computer-using agent (CUA) trajectories. They rigorously labeled diverse agent actions with ground-truth verdicts through multi-stage human annotation, allowing for a more accurate assessment of VLM performance. Additionally, they introduced OSReward-Hard and OSReward-Multi to focus on challenging cases and fine-grained scoring. Their findings indicate that while some VLMs can perform adequately, many still fall short of being reliable judges, particularly when it comes to cost-effectiveness. By releasing OS-Shepherd-100K, a dataset of reasoning-annotated judgments, they provide a pathway for developing more reliable and affordable reward models for CUAs, which could significantly enhance the scalability of AI systems in real-world applications.
The introduction of OSReward and its challenge sets represents a meaningful extension in evaluating VLMs for CUA trajectories.
The systematic evaluation of VLM judges reveals significant shortcomings, backed by rigorous human-annotated ground truth.
Deep reliability assessment
The methodology supports the evaluation of VLM judges on CUA trajectories using a high-quality benchmark, but the claim that these models can fully replace human verification at scale may be overclaimed due to the leniency bias observed in state-of-the-art models.
Reproducibility
yes, the paper mentions that the code, benchmark, dataset, and model checkpoints are available at the OSReward Homepage.
Key figure
Figure 1 illustrates the performance of VLM judges on CUA trajectories and the strict-lenient bias they share.