PhySGA: A Physics-Space Grounded Agentic Framework for Generalizable and Interactive 3D Composition

Imagine you're designing a virtual environment where robots need to assemble complex structures, like a hanging sculpture or a multi-part machine. Current methods often fall short because they either rely on static predictions of object positions or generate images that don't account for the physical interactions between objects. This leads to issues like objects not staying in place due to gravity or failing to fit together correctly, which is known as the problem of physical plausibility in simulations. These failures highlight the limitations of existing approaches that lack a grounded understanding of physics and interactive construction processes. To tackle these challenges, a new framework called PhySGA has been developed. It combines agent exploration with physical simulation, allowing for a more dynamic and realistic assembly of various object types. The framework features an Assembly Agent that plans the overall structure and makes coarse adjustments, while a Critic Agent focuses on the finer details of how objects interact physically. Additionally, Action Routers translate high-level plans into specific actions within the simulation, and a Self-Evolving Skill Library helps the system learn from past experiences to improve future attempts. Compared to previous methods, PhySGA shows significant improvements in how well it maintains the relationship between objects, adheres to physical laws, and produces visually appealing results, making it a valuable tool for anyone working on 3D content creation or robotic simulations.

The approach introduces a novel framework for interactive 3D object composition that integrates physical simulation and agent-based exploration.

The experiments conducted in Isaac Sim provide a solid basis for evaluating the framework's effectiveness against existing methods.