ExecCritic: Learn to Test, Test to Improve for Coding Agents
Leitian Tao, Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, Sharon Li, Jianfeng Gao
Read on arXiv →
Key claim
Separation of roles improves coding agent performance.
In plain English
Imagine you're developing an AI that can automatically fix bugs in software repositories. The challenge lies in ensuring that the tests used to validate these fixes accurately reflect the intended behavior of the code. Often, when an AI generates both the patch and the test, they can share the same mistakes, leading to a false sense of confidence in the solution. This situation is known as test-induced bias, where the tests fail to catch errors because they are not independently verified.
To address this, ExecCritic introduces a structured approach that separates the roles of test generation and code repair. It employs two distinct agents: a Test agent that creates tests independently and a Repair agent that revises the code based on feedback from those tests. This separation allows for more reliable test construction, as the Test agent focuses solely on generating valid tests, while the Repair agent learns to improve code based on the execution results of those tests. The results show that using high-quality tests significantly improves the success rate of code repairs, demonstrating a clear advancement over previous methods that did not utilize this role separation.
ExecCritic introduces a novel separation of roles in coding agents for improved test generation and code repair.
The evaluation on SWE-bench Verified provides solid metrics, though the reliance on specific models may limit generalizability.
Deep reliability assessment
The methodology supports separating test construction from source-code repair to improve coding agent performance, but the effectiveness heavily depends on the quality of the generated tests, which may not always capture the intended behavior accurately.
Reproducibility
yes, open source code is available at the provided GitHub URL.
Key figure
Figure 1 illustrates the separation of test construction from feedback-guided repair, showing the roles of the Test agent and Repair agent in the EXECCRITIC framework.