GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

Imagine you're developing an AI that can play video games like a human, understanding complex instructions and adapting to different gameplay scenarios. The challenge lies in the current benchmarks, which often focus on a limited set of games or lack the depth needed to evaluate real gameplay skills. This leads to issues like not capturing the nuances of long-term planning or the ability to follow multi-step instructions, which are crucial for success in dynamic environments. This is what's called a lack of comprehensive evaluation. To fill this gap, the authors created GameHorizon, a unified suite that includes a large-scale dataset of gameplay recordings, an automated annotation system for multi-horizon instructions, and a benchmark for testing AI models in both offline and online settings. By evaluating 47 different models with over a million invocations, they reveal a clear hierarchy of task difficulty and model performance. This approach not only standardizes how we assess AI in gaming but also provides valuable insights for developers looking to build more capable agents.

GameHorizon introduces a comprehensive evaluation suite for AI in gaming, addressing multiple challenges in existing benchmarks.

The evaluation is based on extensive data collection and rigorous testing across various models, providing a solid foundation for results.