Understanding Reasoning from Pretraining to Post-Training

Imagine you're developing a language model that needs to solve complex reasoning tasks, like playing chess or solving math problems. The challenge lies in understanding how the initial training phase, where the model learns from vast amounts of data, affects its ability to improve through reinforcement learning (RL) later on. Currently, most research treats these two stages separately, making it hard to pinpoint what aspects of pretraining contribute to RL success. This separation can lead to confusion about whether improvements come from the initial training or the subsequent RL phase, which is known as the pretraining-RL interface problem. To tackle this, the authors use chess as a controlled environment to study how different pretraining choices impact RL outcomes. They find that the performance of the model after RL can be predicted based on its pretraining loss, and that the benefits of RL are more pronounced with longer pretraining. Interestingly, RL not only refines the model's existing preferences but also uncovers new strategies that were not apparent during the supervised fine-tuning phase. This approach not only clarifies the relationship between pretraining and RL but also suggests that similar patterns may apply to other domains, like math, indicating a broader relevance for model training strategies. For builders, this means that when designing language models, considering the pretraining phase's impact on subsequent RL could lead to more effective and capable systems.

The paper provides a new framework for understanding the interplay between pretraining and reinforcement learning in language models.

The findings are supported by systematic experiments in a controlled environment, though the transferability beyond chess could be further explored.