Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Training

Imagine you're building a smart assistant that can understand both images and spoken commands, like a robot that can follow your instructions based on what it sees. Currently, when using reinforcement learning (RL) to train these models, the process is quite rigid. It runs in strict phases: first, it gathers data from the environment, then it evaluates that data, and finally, it updates the model. This serial approach can waste a lot of computing power, especially when processing complex visual inputs, because some tasks can be done simultaneously but aren't due to the current setup. This inefficiency is what's called phase-granular execution, and it leads to underutilized resources, particularly in GPU-heavy environments.

To address this, the authors propose a new runtime called Rollplex, which allows for overlapping the computation of visual prompts with the rollout phase of RL. By managing memory and sharing weights more intelligently, Rollplex enables these tasks to run concurrently without disrupting the synchronous nature of RL updates. This results in a notable speedup in processing time, achieving up to 2.24 times faster performance compared to traditional methods. For builders, this means you can train more capable vision-language models more efficiently, making better use of your hardware and potentially speeding up development cycles.

The approach introduces a new runtime optimization for VLMs that enhances efficiency.

The results show significant speedup with a clear methodology, though specific baselines could be more detailed.