Pass the Baton: Trajectory-Relayed On-Policy Distillation

Imagine you're building a machine learning model that generates text based on previous inputs, like a chatbot or a writing assistant. The challenge arises when the model makes a mistake early on; it tends to follow that incorrect path, leading to a cascade of errors in its responses. This issue, known as prefix failure, means that once the model commits to a wrong reasoning direction, it struggles to recover, resulting in unreliable outputs and wasted computational resources. Current methods, like standard on-policy distillation, don't effectively address this problem, leaving models stuck in their erroneous trajectories. To tackle this, Relay On-Policy Distillation (Relay-OPD) introduces a clever mechanism where a teacher model can intervene at critical moments when the student model is likely to go off track. By allowing the teacher to take over briefly and guide the student back on course, Relay-OPD helps the student produce more accurate outputs while still learning from its own experiences. This approach not only improves performance on various mathematical reasoning tasks but also significantly reduces the length of training trajectories, making the training process more efficient. For anyone building AI systems, this means you can achieve better results with less computational overhead, ultimately leading to more reliable and effective models.

The method introduces a novel approach to on-policy distillation that addresses a specific failure mode.

The results show consistent improvements across multiple benchmarks, though the evaluation could be more comprehensive.