Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

Imagine you're creating a video game where characters need to perform realistic movements, like a monkey jumping over a crossbar. Current models can generate impressive videos, but they struggle to translate those movements into usable 3D animations that can be replayed in a game engine. This limitation often leads to either overly complex models that only work with human skeletons or methods that rely on converting 2D poses into 3D motion, which can be inaccurate and cumbersome. This is what's called the 'motion capture problem.'

To address this, the authors propose a model that generates both video and 3D motion simultaneously, allowing for any type of character rig. By keeping the 3D motion generation separate from the video generation, the model can learn how different skeletons move using motion data alone, which is easier to gather than paired video data. This separation also enables the model to accurately locate joints in video frames, facilitating the integration of 3D motion into the scene. Compared to previous methods, this approach allows for more faithful animations from text prompts and better responsiveness to the surrounding environment, making it a significant step forward for developers looking to create dynamic and interactive animations.

The approach separates 3D motion generation from video, allowing for more flexible and realistic animations.

The model is evaluated on a substantial dataset and shows improvements over existing methods.