Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning

Imagine you're building a language model that needs to solve complex reasoning tasks, like advanced math problems. Currently, many models rely on a method called naive repeated sampling, where they generate multiple answers independently and hope one is correct. This often leads to many similar answers that don't explore the problem space effectively, which is a limitation known as local decoding noise. It means that while you might get a lot of attempts, they often don't provide the variety needed to tackle difficult questions. This is what's called ineffective exploration.

To address this, the authors propose a method that first generates a range of relevant concepts or strategies before producing an answer. This approach allows the model to explore the problem space more meaningfully by conditioning the answer generation on these diverse concepts. They even take it a step further by training a small concept generator using reinforcement learning, optimizing it to produce concepts that enhance the performance of a larger, fixed answer generator. The results show that this trained generator significantly improves the model's ability to solve challenging problems compared to traditional methods, even outperforming larger models that weren't specifically tuned for the task. For builders, this means there's potential to create more effective AI systems that can reason better by leveraging a structured exploration of ideas.

The approach introduces a novel way to generate diverse concepts for reasoning tasks.

The evaluation shows improved performance on hard problems, but details on baselines could be clearer.