Reinforcement Learning without Ground-Truth Solutions can Improve LLMs
Yingyu Lin, Qiyue Gao, Nikki Lijing Kuang, Xunpeng Huang, Kun Zhou, Tongtong Liang, Zhewei Yao, Yi-An Ma, Yuxiong He
Read on arXiv →
Key claim
RiVER improves LLM training without ground-truth solutions.
In plain English
Imagine you're trying to teach a model to solve coding problems, but you don't always have the right answers to guide it. Traditionally, models learn from clear, correct solutions, but this limits their ability to tackle more ambiguous tasks where the 'right' answer isn't known. This is a real issue in reinforcement learning, where models rely on feedback from correct answers to improve. When that feedback isn't available, it can lead to problems like scale dominance, where some feedback is too strong and skews the learning process, and frequency dominance, where the model gets stuck on less optimal solutions because they appear more often in the feedback it receives.
The authors of this paper propose a new approach called RiVER, which stands for Ranking-induced VERifiable framework. Instead of needing ground-truth answers, RiVER uses a system of continuous feedback based on how well different solutions rank against each other. This method helps the model learn more effectively by focusing on the best-performing solutions while still considering other valid options. By applying this framework to various coding tasks, they found that RiVER significantly improved the performance of large language models, even on benchmarks that require exact solutions. This means that with the right kind of feedback, models can learn to code better without needing to know the correct answers upfront, which opens up new possibilities for training in more complex environments.
RiVER introduces a new framework for training LLMs without ground-truth solutions, addressing significant challenges in reinforcement learning.
The claims are supported by experiments on multiple benchmarks, showing clear improvements over baselines.
Deep reliability assessment
The paper credibly shows that instance-calibrated, rank-based reward shaping on executable, score-based coding tasks improves both in-domain optimization (ALE-Bench) and transfers modestly to exact-solution code benchmarks. It overreaches in suggesting broad generality beyond such deterministic, verifiable settings and does not establish that reward calibration alone (vs. other confounders like sampling strategy or group size) drives the transfer.
Reproducibility
No code or artifacts link is provided in the text shown. Training data are public AtCoder Heuristic Contest tasks; evaluations on ALE-Bench, LiveCodeBench, and USACO are public, but reproducing results likely requires their specific execution harness, reward calibration implementation, and group-relative RL setup, which are not linked here.
Key figure
Figure 1 contrasts traditional answer-matching verification (binary, gold-dependent) with RiVER’s group-wise, rank-based reward using feasibility checks and objective scores to assign relative rewards without any ground-truth outputs.