PoEM: Predicting RL Outcomes from Existing Policies

Imagine you're developing a foundation model that needs to adapt to new tasks based on different reward signals, like following instructions or aligning with human preferences. Currently, if you want to change the reward function, you often have to retrain the model from scratch, which is not only time-consuming but can also lead to instability in the learning process. This is problematic because it limits flexibility and efficiency, especially when trying to combine multiple reward signals or adapt to new ones. This challenge is known as the inefficiency of reinforcement learning (RL) retraining.

To address this, the authors propose a framework called PoEM, which allows you to predict the outcomes of RL for a new reward function without needing to run the RL training again. The key insight is that if the new reward can be expressed as a combination of existing rewards, the resulting policy can also be derived from the existing policies. Even when the rewards aren't linearly related, the authors found that the policies often lie in a low-rank subspace, making it possible to estimate the new policy using outputs from previously trained models. This approach not only saves computational resources but also provides a more stable way to adapt to new tasks. Compared to traditional methods, PoEM offers a more efficient path for building adaptable AI systems.

The framework introduces a novel way to predict RL outcomes without retraining.

Experimental validation across various rewards supports the method's effectiveness.