MirrorWorld: Taming Video Diffusion Models for Mirror Reflection Generation

Overview§

MirrorWorld is a reflection-aware video inpainting framework built on video diffusion models (VDMs). The key insight is that mirror reflection generation requires solving two complementary sub-problems: what scene content should appear in the mirror and how that content should be spatially arranged. The authors propose two modules—Semantic Relation Distillation (SRD) and Geometric Transformation Alignment (GTA)—that address these sub-problems respectively. They also construct a unified benchmark from four existing video mirror datasets for the reflection reconstruction task.

Semantic Relation Distillation (SRD)§

SRD transfers relational knowledge from a frozen visual foundation model (e.g., DINOv2 or CLIP) to the diffusion model. The goal is to enforce semantic consistency between the visible scene and the mirror region. Formally, let $\mathbf{f}_{scene}$ and $\mathbf{f}_{mirror}$ be feature maps extracted from the visible scene and the mirror region by the teacher model. The teacher's relation matrix is computed as

$$ \mathbf{A}_{teacher} = \sigma\left( \mathbf{f}_{scene} \mathbf{f}_{mirror}^\top \right), $$

where $\sigma$ is a softmax or L2 normalization. During training, the diffusion model's intermediate features are projected into the same space, yielding $\mathbf{A}_{student}$. The SRD loss is

$$ \mathcal{L}_{SRD} = \left\| \mathbf{A}_{student} - \mathbf{A}_{teacher} \right\|_F^2. $$

This encourages the generated mirror content to have the same semantic associations with the surrounding scene as the real reflection.

Geometric Transformation Alignment (GTA)§

GTA models the spatial correspondence between the scene and the mirror. A lightweight geometric predictor produces a transformation matrix $\mathbf{T} \in \mathbb{R}^{3 \times 3}$ (e.g., an affine or homography) that warps scene coordinates to mirror coordinates:

$$ \mathbf{p}_{mirror} = \mathbf{T} \cdot \mathbf{p}_{scene}. $$

The transformation is applied to the diffusion model's positional embeddings or as an attention bias to guide cross-attention between scene and mirror tokens. The GTA loss combines a reconstruction term and a smoothness regularizer:

$$ \mathcal{L}_{GTA} = \left\| \text{warp}(\mathbf{x}_{scene}, \mathbf{T}) - \mathbf{x}_{mirror} \right\|_2^2 + \lambda \left\| \nabla \mathbf{T} \right\|_2^2. $$

Training and Loss§

The total training objective is

$$ \mathcal{L} = \mathcal{L}_{diffusion} + \lambda_{SRD} \mathcal{L}_{SRD} + \lambda_{GTA} \mathcal{L}_{GTA}, $$

where $\mathcal{L}_{diffusion}$ is the standard denoising loss of the video diffusion model. The framework performs video inpainting with a mask covering the mirror region; the visible content serves as context for both SRD and GTA.

Simplified PyTorch Pseudocode§

# Simplified MirrorWorld training step
def train_step(diffusion, teacher, scene_video, mirror_mask, mirror_target):
    # Extract teacher features (frozen)
    with torch.no_grad():
        feat_scene = teacher(scene_video)
        feat_mirror = teacher(mirror_target)
    A_teacher = normalize(feat_scene @ feat_mirror.T)

    # Diffusion forward with noise
    noise = torch.randn_like(mirror_target)
    t = random_timestep()
    z_t = add_noise(mirror_target, noise, t)
    pred = diffusion(scene_video, mirror_mask, z_t, t)

    # SRD: align student relations with teacher
    feat_pred = latent_projection(pred)
    A_student = normalize(feat_pred @ feat_teacher.T)
    loss_srd = mse(A_student, A_teacher)

    # GTA: predict and apply geometric transform
    T = geometric_predictor(scene_video, mirror_mask)
    warped_scene = warp(scene_video, T)
    loss_gta = mse(warped_scene, mirror_target) + smoothness(T)

    loss = diffusion_loss(pred, noise) + lambda_srd * loss_srd + lambda_gta * loss_gta
    loss.backward()
    optimizer.step()

Benchmark and Results§

The benchmark unifies four existing video mirror datasets into a single reflection reconstruction task. Experimental results show that MirrorWorld outperforms image-based reflection generation methods and strong video inpainting baselines, confirming the benefits of explicitly modeling scene-to-mirror semantic and geometric relationships.