DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generation

DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal RAG§

Core Idea§

Multimodal Retrieval-Augmented Generation (MM-RAG) struggles with multi-hop reasoning because traditional methods match independent instances and fail to model explicit relationships across modalities. Graph-enhanced approaches attempt to capture structure but face a dilemma: fine-grained visual features cause graph explosion and retrieval noise, while coarse-grained features discard local evidence. DualG-MRAG resolves this by explicitly decoupling the problem into two complementary graph levels:

• Macro Graph ($\mathcal{G}_{\text{macro}}$): Coarse-grained, global topology for efficient routing among documents and high-level semantic concepts. It suppresses noise by isolating structural reasoning.
• Micro Graph ($\mathcal{G}_{\text{micro}}$): Fine-grained local evidence (e.g., region-level visual features, sentence-level text) for precise verification of candidate evidence.

Methodology§

Graph Construction. Both graphs are constructed from a multimodal knowledge base. Macro nodes represent documents and scene-level visual elements; edges encode semantic similarity or co-occurrence. Micro nodes represent localized visual regions (e.g., detected objects) and textual phrases; edges encode fine-grained cross-modal alignment (e.g., CLIP-based similarity).

Query-Driven Message Passing (GNN Retriever). Retrieval is formulated as iterative propagation of query relevance scores. Given the query embedding $\mathbf{q}$, the GNN updates node representations:

$$ \mathbf{h}_v^{(k+1)} = \text{UPDATE}^{(k)}\left(\mathbf{h}_v^{(k)}, \text{AGG}^{(k)}\left(\left\{\mathbf{h}_u^{(k)} \cdot e_{uv} \,:\, u \in \mathcal{N}(v)\right\}\right)\right) $$

where $e_{uv}$ is a learnable edge weight and $\mathcal{N}(v)$ denotes neighbors. The query is injected as a global bias in the UPDATE function, making propagation query-aware. After $K$ steps, node relevance scores $s_v = \sigma(\mathbf{w}^\top \mathbf{h}_v^{(K)})$ are used to rank evidence.

Dynamic Programming Decoding. The GNN's forward pass tracks which nodes are activated along the propagation paths. Instead of feeding isolated chunks to the LLM, DualG-MRAG extracts explicit reasoning paths via dynamic programming (Viterbi-like decoding) over the score map. This yields coherent structural evidence chains, e.g., an image region → related caption → linked document → final answer.

Training. The model is trained end-to-end with a contrastive loss over relevant/irrelevant evidence pairs, plus a QA loss for answer generation. The GNN retriever and the LLM generator are jointly optimized.

Implementation Sketch§

import torch
import torch.nn as nn
from torch_geometric.nn import MessagePassing

class DualGraphRetriever(nn.Module):
    def __init__(self, hidden_dim):
        super().__init__()
        self.macro_gnn = MacroGNN(hidden_dim)   # coarse-grained
        self.micro_gnn = MicroGNN(hidden_dim)   # fine-grained
        self.fusion = nn.Linear(hidden_dim * 2, 1)

    def forward(self, macro_data, micro_data, query_emb):
        # macro_data: graph with coarse nodes, micro_data: fine-grained graph
        macro_scores = self.macro_gnn(macro_data, query_emb)
        micro_scores = self.micro_gnn(micro_data, query_emb)
        # Combine scores for final ranking
        combined = torch.cat([macro_scores, micro_scores], dim=-1)
        return self.fusion(combined).squeeze(-1)  # logits

class MacroGNN(MessagePassing):
    def forward(self, data, q):
        x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr
        # Node update with query bias
        return self.propagate(edge_index, x=x, edge_attr=edge_attr, q=q)

Results & Impact§

Experiments on multi-hop multimodal QA benchmarks show DualG-MRAG consistently improves evidence recall and answer accuracy over strong baselines (e.g., standard RAG, graph-RAG variants). The decoupled design offers a practical trade-off: global structure-aware routing without sacrificing local visual detail. It opens a new direction for structured reasoning in multimodal retrieval, potentially benefiting tasks like visual question answering, multimodal fact-checking, and knowledge-grounded dialogue.