The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents

Imagine you're building an AI assistant that can handle various tasks, like scheduling meetings or managing emails. You want it to be effective, but adding new skills can sometimes backfire, making the assistant perform worse on tasks it previously handled well. Current evaluations often focus solely on whether the assistant improves overall, but this misses a critical aspect: some skills can lead to regressions, where the assistant fails tasks it used to succeed at. This is what's called regression, and it can happen for several reasons, such as a skill's presence altering the agent's behavior even when not used, or the skill's instructions overriding the agent's natural understanding of inputs and outputs. These issues can lead to persistent failures that aren't addressed by simply adding more skills.

To tackle this, the authors propose a more comprehensive evaluation framework that looks at both the improvements and the regressions caused by procedural skills. They identify three specific modes of regression that should be avoided and emphasize the importance of grounding and verification over just procedural guidance. This approach shifts the focus from merely adding skills to understanding how they interact with the agent's existing capabilities, providing a clearer path for builders to enhance AI performance without introducing new failure modes.

The paper introduces a nuanced evaluation of procedural skills in LLMs, highlighting regressions as a significant concern.

The findings are based on extensive empirical analysis across multiple benchmarks, though some aspects may benefit from further validation.