An Exam for Active Observers
Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger
Read on arXiv →
Key claim
Current MLLMs lack robust active visual observation.
In plain English
Current multimodal large language models (MLLMs) do not effectively engage in active observation, which is crucial for tasks requiring dynamic visual perception. Existing benchmarks fail to measure this capability, leading to misleading assessments of model performance. The introduction of ActiveVision provides a framework to evaluate how well MLLMs can perform tasks that require repeated visual engagement. This is important for builders as it indicates a fundamental limitation in current models and suggests directions for future improvements in model design and training.
Introduces a new benchmark for measuring active observation in MLLMs.
Results are based on evaluations of multiple models against human performance.
Deep reliability assessment
The methodology supports the claim that current MLLMs struggle with tasks requiring active visual observation, as evidenced by their poor performance on the ActiveVision benchmark. However, the paper may overclaim the generalizability of these results to all real-world applications due to the synthetic nature of the test images.
Reproducibility
Yes, the paper mentions a GitHub repository and dataset, indicating that the benchmark and code are available for reproduction.
Key figure
Figure 1 illustrates three tasks designed to test active visual observation: counting separated regions, tracing a path, and comparing contours, highlighting the need for iterative visual perception.