Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

Technical Breakdown§

Problem & Motivation§

Pixel-level image tampering detection aims to localize manipulated regions in images. With modern VLMs (e.g., ChatGPT, Gemini) enabling high-quality edits, detectors must generalize across unseen manipulation distributions. The paper addresses this as a domain generalization (DG) problem, where each VLM defines a new domain.

Method: PIXAR-DG§

The framework builds on a base detector (e.g., PIXAR) with two key strategies:

1. Balanced Minibatch Sampling

Standard training samples tampered and real images uniformly, leading to gradient bias toward either tampered artifacts or clean-image priors. To balance, each minibatch consists of exactly $K$ tampered images and $K$ real images (paired with their ground-truth masks where tampered regions are zero for real images). This ensures gradients from both classes are stable, preventing optimization collapse. Formally, let the minibatch $\mathcal{B} = \mathcal{B}_t \cup \mathcal{B}_r$ where $|\mathcal{B}_t| = |\mathcal{B}_r| = K$. The loss is

$$ \mathcal{L} = \frac{1}{2K} \left( \sum_{x_t \in \mathcal{B}_t} \ell(f(x_t), m_t) + \sum_{x_r \in \mathcal{B}_r} \ell(f(x_r), \mathbf{0}) \right) $$

where $f$ is the detector, $m_t$ the ground-truth tamper mask, and $\ell$ is a per-pixel binary cross-entropy.

2. Late-Injection Strategy

Training on a large base dataset (e.g., synthetic tampered images from various manipulations) until convergence, then fine-tune on a small set of supporting data from new VLM distributions. This avoids overfitting to limited new data while adapting to distribution shifts. The two-stage training is:

1. Train on base data $\mathcal{D}_{base}$ with balanced sampling until validation plateau.
2. Continue training on $\mathcal{D}_{base} \cup \mathcal{D}_{new}$ (with $|\mathcal{D}_{new}| \ll |\mathcal{D}_{base}|$) for a few epochs, using balanced sampling per domain (each minibatch contains equal numbers from base and new).

Implementation Details§

• Architecture: Uses an encoder-decoder (e.g., ResNet-50 + FPN) with a pixel-wise classification head.
• Base data: $\sim$150K images from 4 manipulation types (splicing, copy-move, removal, inpainting).
• New data: 5K images from target VLMs (e.g., GPT-Images-2.0).
• Optimizer: AdamW, lr=1e-4, batch size 32 (16 tampered + 16 real).

Code Snippet (PyTorch-like)§

class BalancedSampler:
    def __init__(self, tampered_dataset, real_dataset, batch_size):
        self.tampered = tampered_dataset
        self.real = real_dataset
        self.batch_size = batch_size // 2

    def __iter__(self):
        tampered_indices = torch.randperm(len(self.tampered))[:self.batch_size]
        real_indices = torch.randperm(len(self.real))[:self.batch_size]
        return zip(tampered_indices, real_indices)

class LateInjectionTrainer:
    def __init__(self, model, base_loader, new_loader, epochs_base, epochs_new):
        self.model = model
        self.base_loader = base_loader
        self.new_loader = new_loader

    def train_base(self):
        for epoch in range(epochs_base):
            for batch in self.base_loader:
                # standard training with balanced sampling
                pass

    def train_with_injection(self):
        # joint training: combine base and new loaders
        for epoch in range(epochs_new):
            for (x_t, x_r, m_t), (x_t_new, x_r_new, m_t_new) in zip(self.base_loader, self.new_loader):
                # concatenate batches, compute loss
                pass

Results§

On four OOD VLMs (GPT-Images-2.0, Gemini-3.1, FLUX.2, Seedream 4.5), PIXAR-DG achieves average gIoU = 67.3% and cIoU = 52.1%, outperforming PIXAR by +26.1% and +26.8% relative improvement. Ablations confirm both strategies are critical.

Summary§

The paper presents a simple yet highly effective DG framework for image tampering detection, with balanced sampling and late-injection as core ingredients. It sets a new state-of-the-art across diverse VLM domains.