The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

Imagine you're developing an AI that needs to plan complex tasks over extended periods, like coordinating multiple actions in a dynamic environment. Current models often struggle because they rely on vast amounts of unstructured data from the internet, which can lead to unpredictable behavior and poor long-term planning. This is particularly evident when agents face situations where they need to remember past actions or adapt to new information, leading to issues like dropped context or compounding errors over time. This phenomenon is known as planning failure, where the agent's performance deteriorates as the task lengthens. To tackle these challenges, the authors propose a structured approach that allows for a more controlled study of how planning abilities can be developed and refined. They introduce a multi-turn environment that systematically examines how agents acquire, shape, and integrate planning skills. By focusing on the quality and format of training data, they find that constructing explicit world models and using techniques like CoT state transition modeling can significantly enhance an agent's ability to generalize over long horizons. They also explore how different post-training methods can shape planning abilities, revealing that some strategies are more effective than others depending on the quality of the data and the task complexity. Ultimately, this work provides a clearer framework for understanding and improving long-horizon planning in AI agents, which is crucial for building more reliable and capable systems in real-world applications.

The approach introduces a controlled environment for studying long-horizon planning, which is a significant extension of existing methods.

The findings are based on systematic studies across multiple stages, though some aspects may lack extensive baseline comparisons.