Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails

Imagine you're building an AI agent that needs to perform specific tasks in a business setting, like managing customer inquiries or processing data. The challenge lies in ensuring that the agent not only understands the tasks but also adapts to the context and tools available. Current methods often rely on fine-tuning models or using advanced architectures, but they can struggle when the model's training doesn't align with the evolving context of the tasks, leading to issues like poor tool selection or inability to recover from errors — this is what's called model-harness misalignment.

To tackle this, the authors propose a method that evolves the agent's operational framework, or harness, while also incorporating expert feedback. The idea is to first develop a harness using a simpler model, which can then be effectively utilized by a more advanced model. However, they found that simply training the weaker model on expert trajectories could backfire, causing performance drops. Instead, they introduce an on-policy expert-correction pipeline that allows the weaker model to maintain its original planning style while benefiting from expert guidance, thus preserving the compatibility between the model and the harness. This approach not only enhances performance but also provides a structured way to evolve both the model and its operational context together, which is crucial for real-world applications.

The approach combines harness evolution with expert correction, addressing a critical gap in agent performance.

The results are based on multiple enterprise tasks, though some performance regressions were noted.