Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

Imagine you're building a GUI agent that needs to interact with various software interfaces, like a virtual assistant that helps users navigate applications. The challenge arises when these agents encounter new interfaces after deployment; they often can't adapt or learn from their mistakes because their parameters are fixed. This limitation leads to issues like failing to understand new layouts or misinterpreting user commands, which is known as the rigidity problem in deployed models. Current methods that try to adapt at test time, like reinforcement learning, often struggle because they can't learn from failed attempts effectively, which is referred to as the exploration failure issue.

To address these challenges, the authors propose a Test-Time Self-Evolving framework that allows agents to improve continuously after deployment without needing human-annotated data. The framework operates in a closed loop, where the agent explores new interfaces, evaluates its actions, reflects on its mistakes, and internalizes the lessons learned. A key innovation is the use of an MLLM-based Reflector for evaluation and a Reflection-Guided On-Policy Self-Distillation method to translate high-level insights into actionable learning. This approach not only enhances the agent's adaptability but also includes a Contrastive Calibration method to ensure that incorrect predictions do not mislead the learning process. Compared to previous work, this framework fills a significant gap in enabling GUI agents to evolve and adapt in real-world scenarios, making them more reliable and effective in dynamic environments.

The framework introduces a novel approach to self-evolving GUI agents post-deployment.

The experiments across six benchmarks provide solid evidence of effectiveness, though more details on baselines would strengthen reliability.