QuoteBench: How Matched Scores Can Hide Command-Path Failures

Imagine you're building a coding assistant that generates Bash commands to automate tasks. The challenge isn't just getting the right command; it's ensuring that the command works correctly in the specific environment where it's executed. Current evaluations often focus on whether the generated command is correct in isolation, missing critical issues that arise when commands are processed through various interfaces or parsers. This can lead to errors that aren't apparent until execution, which is what's called execution transport failure. For instance, a command might look perfect on paper but fail due to how it's interpreted by the system after generation. This is a significant gap in how we assess the capabilities of these AI agents. To address this, the authors propose QuoteBench, a new evaluation framework that rigorously tests command-issuing agents across multiple configurations and tasks. By validating the final state after execution, they can pinpoint where failures occur and how much recovery is possible through model adjustments. This approach reveals that many models perform well in isolation but struggle significantly when faced with real-world execution challenges. For builders, this means that simply relying on matched scores isn't enough; understanding the entire context of command generation and execution is crucial for developing reliable AI agents.

The approach introduces a new evaluation framework that highlights the importance of execution context in command generation.

The findings are based on extensive testing across multiple configurations, though some results may vary significantly.