The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

Imagine you're developing a system that needs to accurately count events in videos, like how many times a ball bounces or how often someone blinks. Current benchmarks often use fixed clips that mix up various factors like event count and visual complexity, making it hard to pinpoint where things go wrong. This leads to issues like failing to recognize transient events, which is a problem known as temporal reasoning failure. When you rely on these benchmarks, you might miss critical insights about how your model performs in real-world scenarios.

To tackle this, the authors propose a method called trace-grounded parametric profiling, which allows for a more controlled evaluation of event counting across different video tasks. By using 2,190 videos with executable event traces, they can assess how well models like Gemini 3.6 Flash perform in counting events based on their frequency and count. Their findings reveal that while the model can handle persistent state transitions reasonably well, it struggles significantly with transient events, showing that the way events are represented can limit a model's ability to access the necessary evidence. This shift in evaluation from just looking at overall accuracy to a more detailed analysis of where models fail provides builders with a clearer understanding of their systems' limitations and helps guide improvements.

The approach introduces a new method for evaluating event counting in videos, addressing limitations of existing benchmarks.

The evaluation is based on a large dataset and provides detailed insights into model performance, though some results are limited.