DOPD: Dual On-policy Distillation
Xinlei Yu, Gen Li, Qingyi Si, Guibin Zhang, Yuqi Xu, Congcong Wang, Shuai Dong, Kaiwen Tuo, Xiangyu Zeng, Kaituo Feng, Qunzhong Wang, Yang Shi, Xiaobin Hu, Xiangyu Yue, Jiaqi Wang, Shuicheng Yan
Read on arXiv →
Key claim
DOPD improves distillation by managing privileged information effectively.
This paper presents a new method called DOPD that improves the process of distillation in machine learning models by using privileged information. It effectively addresses the issue of privilege illusion, leading to better performance in various tasks compared to traditional methods.
In plain English
Imagine you're trying to teach a model to make decisions based on examples it sees, like a student learning from a teacher. In machine learning, we often use a method called distillation, where a 'teacher' model guides a 'student' model to learn from its outputs. However, this can lead to problems when the student gets confused by extra information that it can't fully understand — this is known as privilege illusion. It happens because the student might mix up what it can actually learn from the teacher with what it can't replicate on its own. This confusion can hinder the learning process, especially when only a few pieces of information are truly valuable for making decisions.
To tackle this, the authors propose a new method called DOPD, which stands for advantage-aware dual distillation. The idea is to smartly manage how the student learns from the teacher by adjusting the strength and type of guidance it receives based on its current understanding. This way, the student can focus on the most important signals and avoid getting lost in the noise of irrelevant information. The results show that DOPD consistently outperforms traditional distillation methods, making it a promising approach for improving model training in both language and vision tasks.
For anyone building models, this means you can achieve better performance by using this new method of distillation, especially in complex scenarios where information can be misleading.
The approach introduces a new paradigm for distillation that leverages privileged information to enhance performance.
The claims are supported by extensive experiments across multiple model settings, demonstrating consistent improvements.
Deep reliability assessment
The methodology supports the claim that token-level routing between privileged teacher and privileged self-supervision can outperform vanilla OPD in the authors' Qwen3/Qwen3-VL distillation setups. The broader claim that DOPD generally alleviates "privilege illusion" is plausible but somewhat overclaimed unless the paper directly isolates information-asymmetry effects across more privilege types, model families, and production-like data regimes.
Reproducibility
No code repository is mentioned in the provided abstract, introduction, results, or conclusion excerpts. The paper uses named public benchmarks and includes prompts for privileged input generation, but the provided text is insufficient to confirm full reproducibility details such as training data, exact hyperparameters, checkpoints, or scripts.
Discussion questions
- 1.The paper's core diagnosis is "privilege illusion": gains from privileged hints or annotations may be unlearnable information asymmetry rather than transferable capability. Has anyone here seen a smaller model imitate outputs that looked better during training but failed once the privileged context disappeared?
- 2.DOPD routes token-level supervision based on advantage gap and relative probabilities. Would you trust that routing signal in your own distillation pipeline, or would you worry it is just another noisy heuristic layered on top of already noisy token probabilities?
- 3.The paper compares DOPD against vanilla OPD and several OPD-style counterparts in Qwen3-to-Qwen3 and Qwen3-VL-to-Qwen3-VL settings. Is that a fair enough baseline suite, or would you want to see it against a much simpler recipe like stronger filtering, rejection sampling, or preference optimization without privileged inputs?
- 4.The method deliberately gives different tokens different supervision sources, strengths, and objectives. In a production post-training stack, is that extra complexity worth a roughly 5 to 6 point average gain, or would you prefer a less adaptive but easier-to-debug distillation objective?
- 5.The results show gains on reasoning, coding, function calling, and VLM benchmarks, but all within fairly curated evaluation settings. What is the first missing experiment you would run before adopting DOPD: larger teacher-student gaps, noisy generated privileges, non-Qwen models, or a real downstream product task?
Key figure
Figure 1 compares DOPD with vanilla OPD, teacher/student policies, and other OPD variants across LLM and VLM benchmark suites, showing DOPD with the highest reported average performance in both settings.