Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

Imagine you're building an AI that needs to solve complex problems, like planning a project that requires both math and scheduling skills. The challenge is that most current benchmarks only test individual skills in isolation, which doesn't reflect how these models will perform in real-world scenarios where they need to switch between different types of reasoning. This limitation can lead to failures in understanding context or applying the right skill at the right time, which is what's called a skill-switching gap. To address this, the authors propose a new evaluation framework called Skill^2-Bench, which measures how well models can handle tasks that require multiple skills in a sequence. They introduce a concept called Skill Entropy, which quantifies the difficulty of switching between skills, and use this to create a benchmark with a variety of tasks across different domains. Additionally, they develop a training method called Skill-Entropy RL that encourages models to predict not just answers but also the skills they used, improving their performance significantly on the new benchmark. This approach not only enhances evaluation but also provides a reusable training signal that can be applied to existing datasets, making it a valuable tool for developers looking to build more capable AI systems.

The introduction of Skill Entropy and a new benchmark for cross-skill tasks is a meaningful extension of existing evaluation methods.

The evaluation on multiple models and the use of a clear training signal provide solid evidence of effectiveness.