MIRROR: Learning from the Other View for Multi-Modal Reasoning

Overview§

MIRROR addresses the inconsistency in vision-language models (VLMs) when solving multimodal reasoning problems. The authors observe that VLMs often perform differently on text, image, and combined views of the same problem, revealing complementary reasoning paths. To leverage this, they introduce Modality-Informed Reciprocal Reasoning Optimization (MIRROR), a reinforcement learning approach that uses the best-performing view as a teacher to improve other views via reverse-KL divergence.

Methodology§

Dataset: ODA-Data§

They construct a high-quality paired multimodal geometry dataset with three views per problem:

• Text-dominant: problem statement only.
• Image-dominant: diagram only.
• Combined: both text and image.

MIRROR Framework§

For each problem, MIRROR: 1. Evaluates the model on all views to get reward $r_v$ (e.g., correctness). 2. Selects the best-performing view as the teacher: $v^ = \arg\max_v r_v$. 3. Treats the teacher's output distribution $\pi_{\theta}(y|x_{v^})$ as target. 4. Updates other views by minimizing reverse-KL divergence: $$\mathcal{L}_{\text{MIRROR}} = \sum_{v \neq v^} \mathbb{KL}\left[\pi_{\theta}(y|x_{v^}) \middle\| \pi_{\theta}(y|x_v)\right]$$ This is a form of self-distillation, but optimized via RL to encourage exploration.

Training Algorithm§

MIRROR uses a policy gradient approach. The gradient for a view $v$ is approximated as: $$\nabla_{\theta} \mathcal{L}_v \approx -\mathbb{E}_{y \sim \pi_{\theta}(\cdot|x_{v^*})}\left[\nabla_{\theta} \log \pi_{\theta}(y|x_v)\right]$$ This is equivalent to minimizing reverse-KL when samples are drawn from the teacher.

Implementation Details§

• Base model: a pretrained VLM (e.g., LLaVA).
• Reward: binary correctness on a held-out validation set of ODA-Data.
• Training: iterate over batches, compute teacher via argmax over views, then update other views.

Code Snippet§

import torch
import torch.nn.functional as F

def mirror_loss(model, batch):
    # batch contains inputs for each view: text, image, combined
    views = ['text', 'image', 'combined']
    log_probs = {v: model.get_log_probs(batch[v]) for v in views}  # assumed method
    # Compute rewards (e.g., accuracy) - placeholder
    rewards = {v: compute_reward(log_probs[v]) for v in views}
    # Select teacher view
    teacher_view = max(views, key=lambda v: rewards[v])
    teacher_logits = model(batch[teacher_view])  # forward pass for teacher
    # Compute reverse-KL loss for other views
    loss = 0.0
    for v in views:
        if v != teacher_view:
            student_logits = model(batch[v])
            # Use teacher's distribution as target; sample from teacher
            with torch.no_grad():
                teacher_probs = F.softmax(teacher_logits, dim=-1)
            # reverse KL: KL(teacher || student) = cross_entropy(student, teacher_probs) - H(teacher)
            loss += F.kl_div(F.log_softmax(student_logits, dim=-1), teacher_probs, reduction='batchmean')
    return loss

Results§

MIRROR outperforms standard RL (PPO) on geometry reasoning benchmarks, achieving higher accuracy and more consistent behavior across modalities. The self-supervision from the best view effectively transfers reasoning capability to weaker views without additional human annotations.