Reinforcement Learning without Ground-Truth Solutions can Improve LLMs
Yingyu Lin, Qiyue Gao, Nikki Lijing Kuang, Xunpeng Huang, Kun Zhou, Tongtong Liang, Zhewei Yao, Yi-An Ma, Yuxiong He
Read on arXiv →
Key claim
RiVER improves LLM training without ground-truth solutions.
RiVER allows training large language models on tasks without known correct answers, using a novel reward calibration method. It improves performance on both score-based and exact-solution benchmarks, demonstrating its effectiveness in enhancing coding abilities.
In plain English
Imagine you're trying to teach a model to solve coding problems, but you don't always have the right answers to guide it. Traditionally, models learn from clear, correct solutions, but this limits their ability to tackle more ambiguous tasks where the 'right' answer isn't known. This is a real issue in reinforcement learning, where models rely on feedback from correct answers to improve. When that feedback isn't available, it can lead to problems like scale dominance, where some feedback is too strong and skews the learning process, and frequency dominance, where the model gets stuck on less optimal solutions because they appear more often in the feedback it receives.
The authors of this paper propose a new approach called RiVER, which stands for Ranking-induced VERifiable framework. Instead of needing ground-truth answers, RiVER uses a system of continuous feedback based on how well different solutions rank against each other. This method helps the model learn more effectively by focusing on the best-performing solutions while still considering other valid options. By applying this framework to various coding tasks, they found that RiVER significantly improved the performance of large language models, even on benchmarks that require exact solutions. This means that with the right kind of feedback, models can learn to code better without needing to know the correct answers upfront, which opens up new possibilities for training in more complex environments.
RiVER introduces a new framework for training LLMs without ground-truth solutions, addressing significant challenges in reinforcement learning.
The claims are supported by experiments on multiple benchmarks, showing clear improvements over baselines.
Deep reliability assessment
The paper credibly shows that instance-calibrated, rank-based reward shaping on executable, score-based coding tasks improves both in-domain optimization (ALE-Bench) and transfers modestly to exact-solution code benchmarks. It overreaches in suggesting broad generality beyond such deterministic, verifiable settings and does not establish that reward calibration alone (vs. other confounders like sampling strategy or group size) drives the transfer.
Reproducibility
No code or artifacts link is provided in the text shown. Training data are public AtCoder Heuristic Contest tasks; evaluations on ALE-Bench, LiveCodeBench, and USACO are public, but reproducing results likely requires their specific execution harness, reward calibration implementation, and group-relative RL setup, which are not linked here.
Discussion questions
- 1.Baseline challenge: The main comparison is against RL on raw execution scores. Would a stronger baseline—e.g., per-instance z-score normalization, rank-transformed rewards, or pairwise listwise RLHF-style ranking—close most of RiVER’s gains, and is the paper ducking those?
- 2.Claim stress test: RiVER assumes a deterministic verifier (constraint checks + objective) and comparable instance-level calibration. In your real codegen or agent pipelines, how often do you actually have such clean, deterministic verifiers—and does that limit this method’s usefulness?
- 3.Result scrutiny: ALE rating rank improves 8.9–9.4%, but transfer to exact-solution benchmarks is only +2.4% (LiveCodeBench) and +3.5% (USACO). If you were shipping a coding model, is that transfer big enough to justify the added RL complexity, or would you rather invest in better unit tests/PRMs?
- 4.Tradeoff pick: RiVER emphasizes top-ranked solutions while keeping bounded feedback for other valid candidates to fight ‘frequency dominance.’ Would you push the weighting more aggressively toward the top (risking mode collapse) or increase exploration via entropy/diversity terms (risking slower convergence)?
- 5.Missing eval: They train only on score-based optimization tasks but claim general coding gains. What’s the one stress test you’d run first—math word problems, compiler/formatter robustness, or noisy/partial verifiers—to check whether the calibrated ranking signal really generalizes beyond AHC-style tasks?
Key figure
Figure 1 contrasts traditional answer-matching verification (binary, gold-dependent) with RiVER’s group-wise, rank-based reward using feasibility checks and objective scores to assign relative rewards without any ground-truth outputs.