AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding

Imagine you're building an autonomous vehicle that needs to navigate safely through various driving conditions. The challenge is that while current models can recognize objects and make decisions, they often struggle with understanding complex, real-world incidents — like how to react in a near-accident scenario. This is a significant problem because if a model can't accurately reason about safety-critical situations, it could lead to dangerous outcomes on the road. This is what's called a reliability gap in autonomous driving systems.

Currently, many evaluations focus on basic object recognition or simple decision-making tasks, but they don't adequately test how well these systems can handle the nuances of real-world driving incidents. For example, a model might perform well in clear weather but fail to account for poor visibility or unexpected obstacles. This is where the AUTOPILOT-VQA benchmark comes in. It provides a structured way to assess how well models can answer questions about dashcam footage, focusing on various safety-relevant factors like weather conditions, road layouts, and accident details.

By requiring models to answer grounded questions about both the context of a scene and the specifics of incidents, this benchmark pushes the boundaries of what we expect from vision-language systems in autonomous driving. It moves beyond just recognizing objects to understanding the implications of those objects in dynamic situations. Practically, this means that developers can use AUTOPILOT-VQA to create more interpretable and robust systems that are better equipped to handle the complexities of real-world driving, ultimately leading to safer autonomous vehicles.

This work introduces a new benchmark for evaluating safety-critical reasoning in autonomous driving.

The dataset is structured and covers diverse scenarios, providing a solid basis for evaluation.