Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations
Mehul Damani, Isha Puri, Idan Shenfeld, Jacob Andreas
Read on arXiv →
Key claim
Combines RL with human feedback for better language generation.
In plain English
Imagine you're trying to build a language model that not only gets the right answers but also sounds natural and engaging. You might start with reinforcement learning (RL), which is great for optimizing tasks with clear success metrics, like generating code or solving math problems. However, the challenge is that RL often focuses only on what can be easily measured, like correctness, and ignores the subtler aspects of human communication, such as style and creativity. This can lead to problems like diversity collapse, where the model produces repetitive outputs, or unnatural responses that don't resonate with people. These issues are known as failure modes in RL training.
To address these shortcomings, this paper proposes a new approach that enhances traditional RL methods by incorporating feedback from human demonstrations. The idea is to use an adversarial setup where a generator model learns to produce outputs that not only maximize task accuracy but also align with human-like qualities. A discriminator model is trained alongside the generator to differentiate between human-written and model-generated outputs. This discriminator acts as a guide, providing feedback on aspects that are hard to quantify with simple scores.
The results are promising: in tasks like bug fixing, the new method produces solutions that are not only accurate but also more diverse and human-like compared to previous RL methods. In story generation, it significantly improves the quality of the narratives while still achieving high performance on standard benchmarks. This means that for builders looking to create more engaging and effective language models, this approach offers a scalable way to balance measurable success with the nuanced qualities of human communication.
The approach introduces a novel adversarial framework that enhances reward signals in RL training.
The claims are supported by diverse domain evaluations and clear performance improvements over existing methods.
Deep reliability assessment
The methodology supports the claim that combining verifiable rewards with adversarial learning improves non-verifiable properties while maintaining accuracy. However, the paper may overclaim the generalizability of these results across all domains without extensive testing in varied environments.
Reproducibility
No open source code or dataset URL is mentioned in the paper.
Key figure
Figure 5 illustrates the narrative feature distributions, showing how RLVR diverges from human distributions while VARL better preserves human-like tone and style.