Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Imagine you're trying to improve a large language model's performance after it's been trained. Traditionally, people assume that every part of the model, every layer, needs to be updated equally to get better results. But what if that's not true? What if only a few layers are really responsible for the improvements you see? This paper dives into that question and finds that you can actually get most of the benefits of reinforcement learning by just training one layer instead of the whole model. This is surprising because it goes against the common belief that all layers contribute equally to the model's performance. The authors introduce a new measure called 'layer contribution' to show how much each layer helps with the overall improvement. They tested this across different models and tasks, and consistently found that the middle layers of the transformer architecture are the most important for gaining performance. This means that if you're building or fine-tuning models, you might save time and resources by focusing on just a few key layers instead of trying to update everything at once.

The paper introduces a new understanding of layer contributions in RL training for LLMs.

The findings are supported by systematic analysis across multiple models and tasks.