SABRE: Scalable and Automated Benchmarking of VLMs under Stress

Imagine you're developing a vision-language model that needs to understand complex images and answer questions about them. The challenge lies in creating effective benchmarks that truly test the model's capabilities, especially when existing benchmarks often fail to reveal weaknesses. Current methods can be labor-intensive and may not cover the diverse scenarios that real-world applications encounter, leading to what's called benchmark overfitting — where models perform well on tests but poorly in practical use. This is where SABRE comes in. It automates the creation of stress tests by converting a structured design into a pipeline that generates images and question-answer pairs, ensuring that the tests are both challenging and relevant. By filtering out easy questions and incorporating human validation, SABRE aims to provide a more accurate assessment of how well models can follow visual evidence rather than relying on learned expectations. The framework has been instantiated with various test scenarios, revealing that existing models struggle significantly, with accuracy rates only reaching up to 31.3%. Compared to previous benchmarks, SABRE offers a reusable and adaptable approach to evaluating vision-language models, making it easier for builders to identify and address weaknesses in their designs.

SABRE introduces a novel automated framework for stress-testing vision-language models.

The methodology includes human review and diverse testing scenarios, enhancing reliability.