Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Read on arXiv →
Key claim
Smaller models can outperform larger ones in task planning.
This paper presents a method that enhances the planning capabilities of smaller multimodal language models, achieving better out-of-distribution generalization. The key result shows that their 7B model outperforms a larger 32B model, highlighting the effectiveness of their approach.
In plain English
Imagine you're trying to build a digital assistant that can help you with repetitive tasks across different websites. The challenge is that these tasks can be complex and require careful planning to break them down into manageable steps. Currently, many models struggle with this because they either lack the ability to plan effectively or don't generalize well across different environments. This is particularly problematic when the tasks vary significantly from what the model was trained on, leading to failures in execution — a situation known as poor out-of-distribution (OOD) generalization.
To tackle these issues, the authors propose a method that allows the model to explore different environments and learn from past experiences. This approach helps the model to create high-level training data that is closely aligned with the tasks it needs to perform. They also introduce a framework to analyze how well the model can generalize across different levels of task complexity, revealing that just mastering simple tasks doesn't guarantee success in more complex planning scenarios.
What sets this work apart is that their smaller model, with only 7 billion parameters, achieves a 30.6% accuracy rate, outperforming a much larger model with 32 billion parameters. This suggests that with the right training and experience utilization, smaller models can be just as effective, if not more so, than their larger counterparts in certain applications. For builders, this means that investing in efficient training methods for smaller models could yield significant benefits in real-world task execution.
The paper introduces a new method for task planning in multimodal agents, addressing limitations of existing models.
The claims are supported by experiments on real-world benchmarks, demonstrating improved performance over larger models.
Deep reliability assessment
The methodology supports the claim that hindsight relabeling of autonomous web exploration traces can improve cross-website planning accuracy for small MLLM GUI agents on information-seeking and navigation tasks. It overclaims if read as evidence for general GUI automation, since the evaluation excludes authenticated sessions, CAPTCHA, payments, and other high-risk stateful workflows.
Reproducibility
No code repository is mentioned in the provided paper text. The evaluation uses the public WebVoyager benchmark, but the autonomously explored training trajectories, generated PEEU tasks, and full prompts are only described as being in appendices, not clearly released.
Discussion questions
- 1.The paper’s core move is hindsight relabeling: if the agent searched for a blue 4-star snowboard but actually found a yellow 3-star snowboard, turn that trajectory into a correctly aligned yellow-3-star task. In production, would you trust this as 'turning trash into treasure,' or does it risk training agents to rationalize accidental behavior instead of satisfying original user intent?
- 2.They show a 7B PEEU-trained model reaching 30.6% on WebVoyager and beating the 7.8% instruct baseline, but the tasks are mostly information-seeking and navigation. If your product involves login, forms, payments, admin dashboards, or irreversible actions, does this evaluation gap make the result much less useful, or is planning transfer still the main thing you care about?
- 3.The authors compare against atomic-task training, coarse high-level task training, and experience prompting under the same data scale. Is that the fair fight, or should the real baseline be something like trajectory filtering with a verifier, human-labeled task relabeling, or using a stronger model such as GPT-4o to clean the trajectories?
- 4.Their TDHAF analysis argues that mastering low-level atomic GUI skills does not imply high-level planning competence, while high-level task training generalizes better OOD. For builders here, would you spend limited data budget on high-level task traces first, or on rock-solid action grounding primitives like click, type, scroll, and element selection?
- 5.PEEU depends on autonomous exploration of real websites to discover constraints that are not visible from the homepage, such as ingredients, photos, ratings, or filters. Has anyone tried this kind of exploration in the wild, and did you hit blockers like rate limits, UI randomness, localization, anti-bot defenses, or terms-of-service concerns?
Key figure
Figure 1 contrasts prior atomic or coarse task-to-trajectory training with PEEU’s loop of autonomous exploration, trajectory-to-experience extraction, and hindsight generation of better-aligned high-level tasks.