World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video
Liyuan Zhu, Shengyu Huang, Amrita Mazumdar, Tianye Li, Zan Gojcic, Gordon Wetzstein, Iro Armeni, Shalini De Mello, Alex Trevithick
Read on arXiv →
Key claim
New method improves dynamic 3D reconstruction from monocular videos.
In plain English
Imagine you're trying to create a 3D model of a scene just from a video taken with a single camera. This is tricky because the camera can miss details or create artifacts, especially when the scene is dynamic or the viewpoint changes. Current methods often struggle with these issues, leading to incomplete or inaccurate 3D reconstructions. This is what's called monocular reconstruction failure, where the lack of multiple viewpoints limits the model's understanding of depth and motion.
The approach in this paper tackles these challenges by using a new method that generates dynamic 3D Gaussian representations from monocular videos. It does this by conditioning a video model on detailed, pixel-aligned renderings that capture not just the appearance but also the geometry and motion of the scene. By training on a dataset of aligned multiview video pairs, the model learns to correct common rendering artifacts and fill in missing regions, which are typical problems in monocular reconstruction.
What’s exciting is that this method not only improves the quality of the 3D models but also sets a new state of the art in 4D reconstruction. It can handle real-world videos with significant viewpoint changes and dynamic motions, making it a practical solution for builders looking to create accurate 3D representations from standard video footage.
The method introduces a new way to generate dynamic 3D representations from monocular videos, significantly advancing the state of the art in 4D reconstruction.
The approach is supported by a well-constructed dataset and demonstrates improvements in novel-view synthesis and 3D motion consistency.
Deep reliability assessment
The methodology supports generating dynamic 3D Gaussian representations from monocular videos, but the generalization to all in-the-wild scenarios may be overclaimed without extensive testing across diverse environments.
Reproducibility
No open source code or dataset is mentioned in the paper.
Key figure
Figure 1 illustrates the progression of the World from Motion method, showing how a dynamic 3DGS world is reconstructed from monocular video input through various stages of refinement.