The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric§

Overview§

This paper addresses the limitation of existing perceptual similarity metrics that collapse multiple senses of similarity (e.g., shape, color, texture) into a single scalar. The authors introduce TPIPS (Text-Prompted Image Perceptual Similarity), a metric that conditions similarity judgments on a free-form text prompt describing a specific aspect. They also contribute a large-scale dataset of human similarity judgments over image triplets annotated with multiple semantic aspects.

Dataset: Aspect-Triplet Dataset§

The authors collect human judgments using a three-alternative forced-choice (3AFC) task: given a reference image and two candidates, annotators select which candidate is more similar to the reference under a specified aspect (e.g., "shape", "color"). Each triplet is annotated by multiple annotators across multiple free-form aspects, resulting in 50,000 unique triplets with 420,000 judgments covering 3,000 aspects. The dataset includes both in-domain (from existing datasets like BAM, CUB, Food-101) and out-of-distribution (natural images from LAION-400M) samples.

Model: TPIPS§

TPIPS fine-tunes a pre-trained Vision-Language Model (VLM) — specifically a CLIP-like architecture with a ViT-L/14 vision encoder and a text encoder — to predict the probability that image $I_i$ is more similar to reference $I_r$ than image $I_j$, given a text prompt $T$. The model outputs a similarity score $s(I_r, I_i | T)$ and the probability is computed via a softmax over candidate similarities:

$$ P(i \succ j) = \frac{\exp(s(I_r, I_i | T))}{\exp(s(I_r, I_i | T)) + \exp(s(I_r, I_j | T))} $$

The similarity score is defined as the dot product between a visual representation $v(I)$ and a text-conditioned visual representation $v_T(I)$:

$$ s(I_r, I_i | T) = \langle v(I_r), v_T(I_i) \rangle $$

where $v(I) = \text{ViT}(I)$ and $v_T(I) = \text{ViT}(I) \odot \text{MLP}(\text{TextEnc}(T))$ with element-wise multiplication (gating). The model is trained with a binary cross-entropy loss over triplets.

Training Details§

Initialization: From OpenCLIP ViT-L/14@336px.
Datasets: In-domain (BAM, CUB, Food-101) plus OOD natural images.
Optimization: AdamW with learning rate 1e-5, batch size 128, trained for 10 epochs.
Inference: Given a reference image, a candidate image, and a text prompt, compute $s$ as above.

Code Snippet (PyTorch-like)§

import torch
import torch.nn as nn
from transformers import CLIPModel, CLIPProcessor

class TPIPS(nn.Module):
    def __init__(self, clip_model_name="openai/clip-vit-large-patch14"):
        super().__init__()
        self.clip = CLIPModel.from_pretrained(clip_model_name)
        self.gate = nn.Sequential(
            nn.Linear(768, 768),  # text embedding dimension
            nn.ReLU(),
            nn.Linear(768, 768)
        )

    def forward(self, ref_img, cand_img, text):
        # Get image features
        ref_feat = self.clip.get_image_features(ref_img)  # [B, 768]
        cand_feat = self.clip.get_image_features(cand_img)
        # Text feature for gating
        text_feat = self.clip.get_text_features(text)  # [B, 768]
        gate = torch.sigmoid(self.gate(text_feat))
        # Conditioned candidate feature
        cand_cond = cand_feat * gate
        # Similarity score = dot product
        score = (ref_feat * cand_cond).sum(dim=-1)
        return score

Results§

Human alignment: TPIPS achieves higher 3AFC accuracy (77%) than baselines like CLIP (63%), DINO (62%), and LPIPS (59%) on held-out in-domain data. On OOD data, TPIPS also generalizes well (71% vs. CLIP 59%).
Text-guided retrieval: Using TPIPS as a similarity metric for text-conditioned retrieval improves Recall@1 by 5-10% over CLIP on CUB and BAM.
Compositional search: Demonstrated by retrieving images with specific shape and texture combinations, e.g., "round object with red color".
Fine-grained evaluation: TPIPS better detects mode collapse and diversity in generative models (e.g., StyleGAN, diffusion models) compared to FID and CLIP scores.

Key Contributions§

1. A large-scale dataset of human similarity judgments with multiple free-form aspects. 2. TPIPS: a text-prompted perceptual metric that aligns with human similarity perception across diverse aspects. 3. Demonstrated utility in retrieval, compositional search, and generative model evaluation.

Conclusion§

TPIPS bridges the gap between human perceptual judgments and automated metrics by incorporating contextual text prompts. It sets a new state-of-the-art in aspect-conditioned image similarity and opens up new applications in multimodal understanding.

Abstract

Technical Analysis & Implementation