Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies

Imagine you're building a system that can answer questions about long videos, like a documentary or a day-long recording of events. The challenge is that objects in these videos can be hard to track; they might look similar or be described in the same way, leading to confusion about which object is which. This is problematic because when you ask a question about a specific object, the system might struggle to connect observations of that object across different parts of the video, resulting in incomplete or incorrect answers. This issue is known as identity resolution failure, where the system can't maintain a coherent 'biography' of the object in question. To tackle this, the authors propose a framework called Grounded Entity Biographies (GEB). The idea is to create a memory system that groups observations of the same physical object across different clips, allowing the system to retrieve a comprehensive biography of that entity when answering questions. By linking observations and maintaining context, GEB enables the model to follow an entity through various events, improving the accuracy of responses. Evaluations on several benchmarks, including long-duration recordings, show that GEB outperforms previous memory frameworks, achieving notable accuracy improvements, particularly on the EgoLifeQA dataset. This means that for builders, GEB offers a more reliable way to handle complex video data, making it easier to develop applications that require understanding and reasoning about long-term events.

The approach introduces a novel framework for tracking entity identities across long video sequences.

Evaluation on multiple benchmarks shows solid improvements over existing methods.