TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

Imagine you're developing a system that needs to track multiple objects in a video, like a security camera monitoring a busy street. Current models struggle with a trade-off: they can either follow a few objects over long periods or track many objects but only for short clips. This limitation often leads to missed details or redundant data, which is frustrating when you need comprehensive insights from lengthy footage. This is what's called the tracking trade-off, where the model's effectiveness diminishes based on the duration and number of objects being tracked.

TrackEverything offers a fresh perspective by treating videos as persistent 3D scenes, allowing it to track all visible points across extensive video lengths without the usual memory constraints. The approach is built on three key innovations: it uses a voxelization method to avoid duplicating tracks, separates the tracking process into refining endpoints and classifying points, and introduces a new sampling technique that reduces memory usage. Compared to previous models, TrackEverything not only tracks more points but also significantly outperforms existing dense trackers on shorter clips while maintaining competitive performance on longer sequences. For builders, this means a more efficient and comprehensive tracking solution that can handle complex scenes over extended periods.

TrackEverything introduces a novel approach to 3D point tracking that overcomes existing limitations.

The performance metrics on TAPVid-3D demonstrate solid improvements over existing methods.