PhysiCalWorld: In-Context Physics Calibration for World Models

Imagine you're developing a robot that needs to navigate and interact with various environments, like a home or a factory. The challenge is that traditional models often rely on limited visual input or short histories, which can lead to inaccurate predictions about how objects behave. This can result in issues like the robot misjudging how far to reach or how much force to apply, a problem known as model drift. Without understanding the underlying physics, such as an object's weight or friction, the robot's actions can become unreliable in new situations. To address this, the authors propose a method called PhysiCalWorld, which allows the robot to learn about the specific dynamics of its environment through short interaction clips. Instead of needing extensive physics knowledge or fine-tuning, this method creates a compact descriptor that captures the essential physical properties of the scene. This descriptor can then be reused, making the robot's predictions more accurate and reliable across different tasks. Compared to previous approaches, PhysiCalWorld not only improves prediction accuracy but also creates a structured latent space that generalizes well, meaning it can adapt to new scenarios without extensive retraining.

The approach introduces a novel way to calibrate world models using in-context prompting without traditional physics modeling.

The evaluation across diverse environments shows solid improvements in prediction accuracy, though more extensive baselines could strengthen claims.