ORCA-bench: How Ready Are Language Model Agents for Oncall?

Imagine you're an engineer on call, trying to diagnose a system failure based on a mix of user reports, logs, and metrics. The challenge lies in the ambiguity of the reports and the noise in the data, which can lead to incorrect conclusions. Current tools often fall short because they are tested in simplified environments that don't reflect the complexities of real-world incidents. This is what's called a lack of robustness in production settings, where the stakes are high and the data is messy. To address this, the authors created ORCA-bench, a benchmark designed to evaluate coding agents in a realistic oncall context. It pairs a live microservice system with a comprehensive set of RCA tasks that vary in difficulty and specificity, allowing for a more nuanced assessment of how well these agents can reason through complex scenarios. By scoring agents based on their ability to analyze real telemetry data and source code, ORCA-bench provides a clearer picture of their capabilities and limitations. This approach highlights the significant engineering effort still needed before we can trust these agents in critical production environments.

The introduction of a benchmark for real-world oncall scenarios is a meaningful extension of existing evaluation methods.

The use of curated tasks and expert validation provides solid reliability, though the performance metrics indicate significant room for improvement.