SUN: Persistent Programs For Language-Grounded Control-to-Learning-to-Real Policies

Imagine you're building a robot that needs to perform complex tasks, like assembling furniture or navigating a cluttered room. The challenge lies in ensuring that the robot not only follows commands but also understands the context and objectives of its actions. Currently, many systems rely on hand-crafted rewards and simplistic policies that often lead to suboptimal performance, especially in long-horizon tasks. This can result in behaviors that drift away from the intended goals, a problem known as reward hacking or misalignment.

To address this, the authors propose a system called Kuafu, which uses Semantically UNified Programs. This approach allows for the definition of geometric and contact relationships in a way that can be directly translated into control objectives and learning rewards. By leveraging large vision-language models, Kuafu can automatically generate these programs from natural language descriptions and scene semantics. This means that the robot can learn to perform tasks without needing extensive demonstrations or manually defined rewards. In tests across nine different tasks, Kuafu significantly outperformed traditional methods, achieving higher success rates and more efficient trajectory generation. This shift towards integrating symbolic planning with data-driven execution could fundamentally change how we approach robotic learning and control.

The introduction of Semantically UNified Programs represents a significant advancement in integrating control and learning in robotics.

The results are backed by comprehensive experiments across multiple tasks, though some metrics could be more robust.