DemoPSD: Disagreement-Modulated Policy Self-Distillation
Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
Read on arXiv →
Key claim
DemoPSD reduces information leakage and enhances exploration.
DemoPSD effectively reduces privileged information leakage while preserving exploration in language model training. This leads to better generalization on out-of-distribution tasks, making it a valuable tool for builders working with large language models.
In plain English
Imagine you're training a language model to answer questions across various topics. You want it to learn effectively from its own experiences while also benefiting from guidance provided by a teacher model. However, if the teacher model gives too much specific information, the student might overfit to those details and struggle when faced with new, unseen questions. This is a common issue known as privileged information leakage, where the student learns shortcuts that don't apply in real-world scenarios. Additionally, the student might lose its ability to explore different reasoning paths, which is crucial for tackling diverse problems.
Currently, many approaches rely on dense supervision from the teacher, which can lead to these pitfalls. The paper introduces a new method called DemoPSD, which allows the student to learn from the teacher without fully relying on its guidance. Instead of just mimicking the teacher's outputs, DemoPSD encourages the student to maintain its own reasoning abilities while selectively adopting useful insights from the teacher. This is achieved through a clever mechanism that balances the learning from both models, ensuring that the student doesn't just memorize answers but also develops a deeper understanding.
The results show that DemoPSD not only mitigates the leakage of privileged information but also preserves the student's capacity to explore different solutions. In practical terms, this means that when you're deploying language models in real-world applications, you can expect them to perform better on new types of questions, thanks to the improved training approach. This is particularly important for applications that require robust generalization across various domains.
The approach introduces a new framework for balancing teacher and student learning in self-distillation, addressing significant issues in existing methods.
The claims are supported by extensive experiments across multiple domains, demonstrating solid performance improvements.
Deep reliability assessment
The methodology supports the claim that DemoPSD can mitigate privileged information leakage and preserve exploration capacity, but the generalization claims may be overextended without testing under different types of distribution shifts like label shift.
Reproducibility
No open source code or dataset URL is mentioned in the paper.
Discussion questions
- 1.The authors claim DemoPSD effectively mitigates privileged information leakage. How often do you encounter this issue in your own work, and do you think their approach would address it effectively?
- 2.DemoPSD is compared against GRPO and SDPO. Do you think these are the most appropriate baselines, or is there another method that would provide a more rigorous comparison?
- 3.The paper focuses on in-domain and out-of-domain generalization. Have you experienced similar challenges in your projects, and how did you address them?
- 4.DemoPSD maintains higher training entropy compared to SDPO. Do you think this is a beneficial tradeoff, or could it lead to other issues in model performance?
- 5.The authors use a reverse-KL barycenter target for balancing teacher and student distributions. How feasible is this approach in real-world applications, and what challenges might arise?
Key figure
Figure 1 illustrates that DemoPSD maintains higher policy entropy over training steps compared to SDPO, which translates into better best@16 performance across domains.