SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na
Read on arXiv →
Key claim
SWE Refactor Bench rigorously evaluates coding agent migrations.
In plain English
Imagine you're maintaining a large software system that has accumulated years of technical debt. When it comes time to migrate to a new architecture or language, the process can be daunting and often requires manual intervention. Current benchmarks for coding agents primarily check if the code behaves correctly after a migration, but they don't verify if the migration itself was actually completed. This leads to a situation where agents might simply copy the old code to pass tests, a problem the authors refer to as Blindness. This oversight can result in significant issues down the line, as the underlying technical debt remains unaddressed. To tackle this, the authors developed SWE Refactor Bench, a benchmark designed to assess coding agents on their ability to perform whole-repository migrations. It includes 20 different migration tasks that cover various types of technical debt. The evaluation process is threefold: first, it checks if the migration was completed, then it tests for behavioral correctness, and finally, it uses independent coding agents to identify any hidden behavioral differences. The results show that only a small percentage of runs successfully pass all stages, highlighting the distinct challenges of migration completeness versus behavioral correctness. This benchmark provides a more comprehensive framework for developing and evaluating coding agents, ensuring they can handle the complexities of real-world software migrations.
Introduces a new benchmark for evaluating coding agents on migration tasks.
Employs a rigorous three-stage evaluation protocol with diverse models and tasks.
Deep reliability assessment
The methodology supports evaluating coding agents' ability to perform whole-repository migrations by measuring both migration completeness and behavioral correctness. However, it may overclaim the general applicability of these results to all migration projects, as the benchmark focuses on specific tasks and configurations.
Reproducibility
yes, the dataset and benchmark details are available on the project homepage.
Key figure
Figure 1 illustrates the three-stage evaluation process of SWE Refactor Bench, contrasting it with prior benchmarks that use a single-stage evaluation.