SceneBind: Binding What and Where Across Vision, Audio and Language

Technical Breakdown§

SceneBind addresses the lack of explicit spatial structure in omni-modal encoders by representing scenes as a combination of a global semantic embedding and object-centric semantic-spatial slots. The model is compatible with large-scale pretrained semantic encoders (e.g., CLIP, AudioCLIP) and adds lightweight spatial modeling with only a few extra tokens.

Core Methodology

Scene Representation: Each scene is encoded into a set of $N$ object-centric slots $\{\mathbf{s}_i\}_{i=1}^N$, where each slot is a tuple $\mathbf{s}_i = (\mathbf{e}_i, \mathbf{p}_i, \sigma_i)$ consisting of a semantic embedding $\mathbf{e}_i \in \mathbb{R}^d$, a 3D spatial position $\mathbf{p}_i \in \mathbb{R}^3$ (with uncertainty modeled via a Gaussian variance $\sigma_i$), and a global scene embedding $\mathbf{g} \in \mathbb{R}^d$. The slots are obtained via a transformer decoder that attends to per-modality features from pretrained encoders.

SceneBind Matching: For cross-modal retrieval, a similarity score between two scenes $X$ and $Y$ (possibly from different modalities) is computed as: $$S(X,Y) = \alpha \cdot \cos(\mathbf{g}_X, \mathbf{g}_Y) + (1-\alpha) \cdot \frac{1}{N} \sum_{i=1}^N \max_j \left( \text{GroundingScore}(\mathbf{s}_i^X, \mathbf{s}_j^Y) \right)$$ where $\alpha \in [0,1]$ balances global and object-level similarity. The GroundingScore uses a combination of semantic cosine similarity and spatial Gaussian log-likelihood: $$\text{GroundingScore}(\mathbf{s}_i, \mathbf{s}_j) = \cos(\mathbf{e}_i, \mathbf{e}_j) + \beta \cdot \log \mathcal{N}(\mathbf{p}_i | \mathbf{p}_j, \sigma_j^2 \mathbf{I})$$

Training: The model is trained with a contrastive loss across modalities (vision, audio, language) using the scene similarity scores, plus a localization loss that encourages object slots to predict correct 3D positions from audio or text queries.

Implementation Details

• Pretrained encoders: ViT-L/14 for vision, CLAP for audio, and text encoder from CLIP.
• Additional spatial decoder: 4-layer transformer with 8 attention heads, outputting slots of dimension 512.
• Dataset: 100k real-world binaural audio-visual scenes with 3D object annotations.

Code Snippet (PyTorch-like)

class SceneBind(nn.Module):
    def __init__(self, d_model=512, n_slots=16, pretrained_encoders):
        super().__init__()
        self.vision_enc = pretrained_encoders['vision']
        self.audio_enc = pretrained_encoders['audio']
        self.text_enc = pretrained_encoders['text']
        self.slot_decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model, nhead=8), num_layers=4)
        self.global_proj = nn.Linear(d_model, d_model)
        self.slot_proj = nn.Linear(d_model, d_model + 3 + 1)  # e, p, sigma

    def forward(self, vision, audio, text):
        # Extract per-modality features (e.g., [CLS] tokens)
        v_feat = self.vision_enc(vision).last_hidden_state[:,0]
        a_feat = self.audio_enc(audio).last_hidden_state[:,0]
        t_feat = self.text_enc(text).last_hidden_state[:,0]
        # Concatenate and decode slots
        query = torch.randn(1, self.n_slots, self.d_model).repeat(v_feat.shape[0],1,1)
        memory = torch.stack([v_feat, a_feat, t_feat], dim=1)
        slots = self.slot_decoder(query, memory)  # (B, n_slots, d)
        # Project to semantic + spatial
        out = self.slot_proj(slots)
        e = out[..., :d_model]
        p = out[..., d_model:d_model+3]
        sigma = out[..., d_model+3:].exp().clamp(min=1e-6)
        global_emb = self.global_proj(slots.mean(dim=1))
        return global_emb, (e, p, sigma)

Key Results

• State-of-the-art on scene retrieval benchmarks (e.g., +5% R@1 on AudioSet scenes).
• Zero-shot audio-visual localization: predicts 3D positions of sound sources from audio alone with 0.4m error.
• Ablations show both global and object-level matching contribute to performance.