Visual Contrastive Self-Distillation

Technical Breakdown of VCSD§

Core Methodology§

Visual Contrastive Self-Distillation (VCSD) is an on-policy self-distillation method for vision-language models (VLMs). It eliminates the need for external teachers, privileged answers, or visual evidence by creating asymmetry purely through input conditioning. The key idea is to use an Exponential Moving Average (EMA) teacher that generates two next-token distributions for each student-generated prefix: one conditioned on the original image ($p_{\text{orig}}$) and one on a content-erased control image ($p_{\text{erased}}$). The token-wise log-probability difference $\Delta = \log p_{\text{orig}} - \log p_{\text{erased}}$ highlights candidates whose likelihood is specifically increased by the actual visual content. This contrastive signal sharpens the teacher's original-image distribution within its plausible support (e.g., top-$k$ tokens):

$$\tilde{p}(y_t) = \text{softmax}\left( \log p_{\text{orig}}(y_t) + \alpha \cdot \Delta(y_t) \right) \cdot \mathbb{1}_{y_t \in \text{support}}$$

where $\alpha$ is a scaling hyperparameter. The resulting distribution $\tilde{p}$ serves as the distillation target, and the student is trained by minimizing the KL divergence between its own logits and this target.

Implementation Details§

• EMA Teacher: Updated as $\theta_{\text{EMA}} \leftarrow \tau \theta_{\text{EMA}} + (1-\tau) \theta_{\text{student}}$.
• Content Erasure: Images are processed to remove instance-level visual content while preserving layout and low-level features (e.g., using random masking or a pretrained erasure module).
• Training Procedure: For each (image, prompt) pair, the student generates a response prefix (on-policy). At each token position, the teacher computes logits under both conditions. The contrastive target is constructed, and the student is updated via KL divergence.
• Evaluation: On ViRL39K benchmark, VCSD improves aggregate scores across 7 benchmarks for Qwen3-VL (e.g., 2B: 62.27% -> 67.04%) and Qwen3.5 models, without additional inference cost.

PyTorch-style Code Snippet§

def train_step(student, ema_teacher, image_orig, image_erased, prompt):
    # Student generates response prefix (on-policy)
    with torch.no_grad():
        prefix = student.generate_prefix(prompt, image_orig)
    
    # Teacher produces distributions for both conditions
    with torch.no_grad():
        logits_orig = ema_teacher(prompt, image_orig, prefix)  # (batch, vocab)
        logits_erased = ema_teacher(prompt, image_erased, prefix)
    
    # Compute contrastive difference
    log_probs_orig = F.log_softmax(logits_orig, dim=-1)
    log_probs_erased = F.log_softmax(logits_erased, dim=-1)
    delta = log_probs_orig - log_probs_erased  # (batch, vocab)
    
    # Build support: top-k tokens from original distribution
    support_probs, support_ids = torch.topk(log_probs_orig.softmax(-1), k=50, dim=-1)
    # Add contrastive signal and renormalize
    target_logits = logits_orig.gather(-1, support_ids) + alpha * delta.gather(-1, support_ids)
    target_probs = F.softmax(target_logits, dim=-1)
    
    # Student logits on same prefix
    student_logits = student(prompt, image_orig, prefix)
    student_log_probs = F.log_softmax(student_logits, dim=-1)
    kl_loss = F.kl_div(student_log_probs.gather(-1, support_ids), target_probs, reduction='batchmean')
    
    return kl_loss

Key Advantages§

• Removes external teacher, privileged answers, and visual evidence signals.
• Simple contrastive mechanism relies only on input conditioning asymmetry.
• Consistent gains across model scales and architectures without extra inference cost.