GROW$^2$: Grounding Which and Where for Robot Tool Use

Imagine you're building a robot that needs to perform tasks in unpredictable environments, like cutting a cake without a knife. Traditionally, robots are limited to using tools as they were designed, which means they struggle when faced with novel situations or objects. For instance, if a robot only knows how to use a knife for cutting, it won't think to use a plate creatively, leading to failures in task execution. This limitation is known as open-world affordance grounding, where the robot must identify and localize the right tool and its parts for a given task.

What this paper introduces is a method called GROW², which helps robots overcome these challenges by breaking down the process of tool use into two levels: semantic and geometric. Semantically, it uses the reasoning capabilities of Vision-Language Models to interpret task instructions and select the right object as a tool. Geometrically, it then pinpoints the specific parts of the tool and the target object that are relevant for the task, all from a single RGB-D image. This hierarchical approach allows the robot to use objects in ways that weren't explicitly trained, making it more adaptable.

Compared to previous methods, GROW² shows significant improvements in predicting how objects can be used as tools, achieving better performance on established benchmarks and demonstrating zero-shot generalization. For builders, this means you can create robots that are not only more capable but also require less retraining when faced with new tasks or objects, ultimately leading to more efficient and versatile robotic systems.

The approach introduces a novel hierarchical method for open-world affordance grounding, leveraging commonsense reasoning and geometric grounding.

The experiments demonstrate strong performance against state-of-the-art baselines across multiple benchmarks, supporting the claims made.