DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generation

Imagine you're building a system that needs to pull together information from various sources, like text and images, to answer complex questions. The challenge is that current methods often treat each piece of information in isolation, which can lead to missing connections between different data types. This is particularly problematic in tasks that require multi-hop reasoning, where you need to link multiple pieces of evidence to arrive at a conclusion. When trying to incorporate visual features, existing graph-based methods struggle with either too much noise from fine details or losing important context with broader representations. This is what's called the dilemma of balancing granularity in multimodal scenarios.

To tackle this, the authors propose a dual-tier framework called DualG-MRAG, which separates the reasoning process into two parts: a Macro Graph for overarching structural reasoning and a Micro Graph for detailed evidence matching. This separation helps reduce noise and allows for more precise retrieval of relevant information. They also introduce a message-passing mechanism that dynamically connects different sources of evidence, enhancing the system's ability to reason across modalities. By providing a structured way to guide the generative model, this approach improves the coherence of the responses generated. Compared to previous methods, DualG-MRAG shows significant improvements in both retrieving relevant evidence and accurately answering complex questions, making it a more reliable tool for building advanced multimodal systems.

The dual-tier framework introduces a novel approach to multimodal reasoning.

Extensive experiments validate the method against established baselines.