Scalable Visual Pretraining for Language Intelligence

Overview§

This paper investigates visual pretraining directly on raw document images (e.g., PDF pages, web screenshots) as an alternative to text-only pretraining for language models. The core idea is that visual signals—such as layout, typography, and embedded figures—carry information that is lost when documents are stripped to plain text. Using the same underlying corpora, visual pretraining achieves superior performance on downstream language tasks, suggesting that multimodal representations are more effective for language intelligence than text alone.

Methodology§

Visual pretraining is performed using a Vision Transformer (ViT) backbone or a hybrid architecture that processes document images at high resolution. The model is trained with self-supervised objectives adapted from computer vision, such as:

• Masked Image Modeling (MIM): Random patches of the document image are masked, and the model predicts the pixel values or a discretized representation (e.g., via VQ-VAE). The loss is a reconstruction loss:

$$\mathcal{L}_{\text{MIM}} = \mathbb{E}_{x \sim \mathcal{D}} \left[ \sum_{i \in \text{masked}} \| x_i - \hat{x}_i \|^2 \right]$$

• Contrastive Learning: Two augmentations of the same document image are encoded, and the model maximizes agreement between their representations while pushing apart different documents. The InfoNCE loss is used:

$$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp( \text{sim}(z_i, z_j)/\tau )}{\sum_{k=1}^{2N} \mathbb{1}_{[k \neq i]} \exp( \text{sim}(z_i, z_k)/\tau )}$$

The pretrained visual encoder is then combined with a small text decoder (e.g., a single-layer transformer) and fine-tuned on language tasks, or used as a feature extractor for downstream models.

Implementation Details§

• Architecture: A ViT-Large (patch size 16, 24 layers) is used as the visual backbone. Input images are resized to 224×224 or higher (e.g., 448×448) to capture fine-grained details like small equations.
• Pretraining Data: A large-scale corpus of document images (e.g., from CommonCrawl, arXiv PDFs) is collected. No OCR or text extraction is performed.
• Training: The model is trained for 500k steps with a batch size of 4096 using AdamW optimizer, learning rate 1e-3, cosine decay schedule.
• Fine-tuning: The pretrained encoder is frozen or partially fine-tuned while a lightweight decoder is trained on downstream datasets (e.g., GLUE, SQuAD).

Code Snippet§

import torch
import torch.nn as nn
import torchvision.transforms as T
from timm.models.vision_transformer import VisionTransformer

class VisualPretrainModel(nn.Module):
    def __init__(self, img_size=224, patch_size=16, embed_dim=1024, depth=24):
        super().__init__()
        self.encoder = VisionTransformer(
            img_size=img_size,
            patch_size=patch_size,
            embed_dim=embed_dim,
            depth=depth,
            num_heads=16,
        )
        # Decoder for masked image modeling
        self.decoder = nn.Sequential(
            nn.Linear(embed_dim, 512),
            nn.GELU(),
            nn.Linear(512, patch_size * patch_size * 3)  # reconstruct pixels
        )
        self.mask_token = nn.Parameter(torch.randn(1, 1, embed_dim))
        
    def forward(self, x, mask_ratio=0.75):
        # x: (B, C, H, W)
        patches = self.encoder.patch_embed(x)  # (B, N, D)
        B, N, D = patches.shape
        # Generate random mask
        len_keep = int(N * (1 - mask_ratio))
        noise = torch.rand(B, N, device=x.device)
        ids_shuffle = noise.argsort(dim=1)
        ids_keep = ids_shuffle[:, :len_keep]
        ids_restore = ids_shuffle.argsort(dim=1)
        
        # Mask tokens
        mask_tokens = self.mask_token.repeat(B, N - len_keep, 1)
        x_ = patches.gather(1, ids_keep.unsqueeze(-1).expand(-1, -1, D))
        x_masked = torch.cat([x_, mask_tokens], dim=1)
        x_masked = x_masked.gather(1, ids_restore.unsqueeze(-1).expand(-1, -1, D))
        
        # Add positional embeddings (not shown for brevity)
        features = self.encoder.blocks(x_masked)
        decoded = self.decoder(features)
        return decoded

# Usage
transform = T.Compose([T.Resize((224, 224)), T.ToTensor()])
model = VisualPretrainModel()
image = torch.rand(2, 3, 224, 224)
output = model(image)  # reconstruct masked patches
loss = nn.MSELoss()(output, image)  # simplified

Results§

Across multiple benchmarks (GLUE, SQuAD, etc.), visual pretraining outperforms text-only baselines by 2–5% on average, especially on tasks requiring reasoning over layout or understanding of figures. The method scales well: larger visual backbones yield further gains, and the approach is data-efficient since it avoids expensive text extraction pipelines.