Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

Imagine you're building a system that processes images and text together, like a smart assistant that can understand both what you say and what you show it. To make these systems faster, you might want to reduce the amount of image data they process by removing unnecessary parts. However, if you cut too much, you risk losing important details that help the system understand your instructions, especially when they are complex or very specific. This is a common issue with current methods of visual token pruning, where the selection process can overlook critical visual information due to noise in the text or poor selection strategies. This is what's called feature fragmentation and textual noise corruption.

To tackle these problems, the authors propose a new approach that first measures the relevance of different parts of the text to the image data, filtering out irrelevant noise. Then, instead of just picking the top few image patches, they use a more sophisticated method that ensures the selected patches work well together and cover the important aspects of the image. This method, called Entropy-Aware Dense Pruning (EADP), helps maintain a complete and useful visual representation.

The results show that EADP not only speeds up the processing of visual language models but also keeps the accuracy high, even when working with limited resources. This means that for anyone building applications that rely on understanding both images and text, this new method could lead to more efficient and effective systems.

The paper introduces a new framework for visual token pruning that addresses specific failure modes in existing methods.

The claims are supported by extensive experiments demonstrating improved performance on multimodal benchmarks.