Learning Native Reflection in Unified Models with Interleaved Reinforcement Learning

Imagine you're building a system that can generate and refine images based on user feedback. The challenge lies in ensuring that the model can effectively learn from its mistakes and improve its outputs over time. Current methods often rely on supervised fine-tuning or reinforcement learning that focuses on either the image generation or the feedback process separately. This can lead to missed opportunities for improvement, as the model may not fully leverage the information from both roles, resulting in what's known as inefficient credit assignment.

To address this, the authors propose a method called UMM-Reflection, which allows the model to learn from multiple reflection trajectories simultaneously. By sharing an initial image across sibling trajectories, the model can compare different strategies for improvement and update its understanding of both the reflection tokens and the image revisions in a more integrated manner. This approach avoids the pitfalls of traditional methods by enabling credit to flow across multiple rounds of feedback, ultimately leading to better performance without needing an external verifier during inference. Compared to previous techniques, UMM-Reflection shows significant gains across various benchmarks, indicating a more effective way to refine multimodal outputs in real-world applications.

The approach introduces a novel way to optimize reflection trajectories in multimodal models.

The improvements are backed by multiple benchmarks, though some metrics may not be widely recognized.