Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
Wenxiao Wang, Priyatham Kattakinda, Soheil Feizi
Read on arXiv →
Key claim
RELAI-VCL enables compounding optimization gains in agents.
In plain English
Current agent optimization methods often report one-time gains without considering ongoing task evolution. This paper identifies that many existing methods fail to maintain improvements when faced with new challenges. By introducing a continual learning framework, the authors demonstrate that their approach, RELAI-VCL, can sustain and even enhance performance over time. Builders might find this relevant as it suggests a pathway to developing more resilient agents that adapt to changing environments.
Introduces a new approach to continual learning in agent optimization.
Evaluates multiple methods under controlled conditions with clear metrics.
Deep reliability assessment
The methodology supports the claim that optimization gains can compound when regression control is integrated into the optimization loop, but it may overclaim the generalizability of these results across different settings without further validation.
Reproducibility
yes, the paper mentions that baseline, Phase 1, and Phase 2 agent artifacts for all three optimizers are released at https://github.com/relai-ai/Continual-Learning-Terminal-Bench.
Key figure
Figure 1 shows the lifelong average pass rate by agent, highlighting RELAI-VCL's superior performance compared to GEPA, Meta Harness, and the unoptimized baseline.