3D-Aware VLMs with Implicit and Explicit Geometries

Technical Synopsis§

Overview§

VLM-IE3D extends standard vision-language models (VLMs) with 3D spatial awareness by introducing two new token types derived purely from RGB videos: Implicit Geometry Tokens (IGTs) and Explicit Geometry Tokens (EGTs). These tokens are fused with 2D visual features via a lightweight 3D-aware adapter, enabling strong performance on 3D detection, grounding, captioning, and spatial reasoning.

Implicit Geometry Tokens (IGTs)§

IGTs capture high-level geometric priors from the input video. A video encoder (e.g., a transformer) processes a sequence of frames to produce frame-level features. These are pooled and projected to a set of learnable implicit tokens that attend to the video features via cross-attention. Formally, let $\mathbf{V} \in \mathbb{R}^{T \times H \times W \times C}$ be the video features. A set of $N_i$ implicit query tokens $\mathbf{Q}_i \in \mathbb{R}^{N_i \times d}$ interact with $\mathbf{V}$:

$$\mathbf{IGT} = \text{CrossAttn}(\mathbf{Q}_i, \mathbf{V}, \mathbf{V})$$

where CrossAttn is multi-head cross-attention.

Explicit Geometry Tokens (EGTs)§

EGTs encode detailed geometric structures from reconstructed 3D attributes. Given the same video, a monocular depth estimator (e.g., MiDaS) is used to obtain depth maps. These are lifted to point clouds and voxelized. A 3D sparse encoder (e.g., MinkowskiEngine) processes the voxel grid to produce geometric features. A set of $N_e$ explicit query tokens $\mathbf{Q}_e$ attend to these features:

$$\mathbf{EGT} = \text{CrossAttn}(\mathbf{Q}_e, \mathbf{F}_{3D}, \mathbf{F}_{3D})$$

where $\mathbf{F}_{3D}$ is the output of the 3D encoder.

3D-Aware Adapter§

The adapter fuses IGTs, EGTs, and 2D visual features. It consists of a transformer decoder that takes three sets of tokens: the original 2D tokens from a CLIP image encoder, IGTs, and EGTs. They are concatenated and processed by self-attention and cross-attention layers that mix the information. The output is then fed into the VLM’s text decoder. The fusion can be summarized as:

$$\mathbf{Z} = \text{Concat}(\mathbf{T}_{2D}, \mathbf{IGT}, \mathbf{EGT})$$ $$\mathbf{Z}' = \text{TransformerDecoder}(\mathbf{Z})$$

Training§

VLM-IE3D is trained end-to-end on video-language datasets with losses for the target tasks (e.g., detection loss, captioning loss). The video encoder and depth estimator are frozen; only the token generators, adapter, and the VLM’s text decoder are trainable.

Code Snippet§

Below is a simplified PyTorch illustration of the model's forward pass.

import torch
import torch.nn as nn
from transformers import CLIPVisionModel

class VLM_IE3D(nn.Module):
    def __init__(self, num_implicit=8, num_explicit=8, d_model=768):
        super().__init__()
        self.video_encoder = ...  # e.g., TimeSformer
        self.depth_estimator = ...  # frozen MiDaS
        self.point_cloud_encoder = ...  # 3D sparse conv
        self.implicit_queries = nn.Parameter(torch.randn(1, num_implicit, d_model))
        self.explicit_queries = nn.Parameter(torch.randn(1, num_explicit, d_model))
        self.cross_attn = nn.MultiheadAttention(d_model, 8)
        self.fusion_decoder = nn.TransformerDecoder(...)  # with self-attn
        self.text_decoder = ...  # from Qwen or LLaMA

    def forward(self, rgb_frames):
        B, T, C, H, W = rgb_frames.shape
        # Video features
        vid_feat = self.video_encoder(rgb_frames)  # (B, T, h, w, d)
        # Implicit tokens from video
        imp_tokens = self.cross_attn(self.implicit_queries.expand(B,-1,-1), 
                                     vid_feat.view(B,-1,d), vid_feat.view(B,-1,d))[0]
        # Depth and 3D features
        depths = self.depth_estimator(rgb_frames[...,0,:,:])  # (B, T, H, W)
        pcd = depth_to_pointcloud(depths)  # (B, N, 3)
        voxel = pointcloud_to_voxel(pcd)  # (B, D, H', W', L')
        feat_3d = self.point_cloud_encoder(voxel)  # (B, d)
        # Explicit tokens from 3D
        exp_tokens = self.cross_attn(self.explicit_queries.expand(B,-1,-1), 
                                     feat_3d.unsqueeze(1), feat_3d.unsqueeze(1))[0]
        # Concatenate with 2D tokens (from CLIP image encoder)
        img_tokens = self.clip_encoder(rgb_frames[:,0,:,:,:])[0]  # first frame
        combined = torch.cat([img_tokens, imp_tokens, exp_tokens], dim=1)
        fused = self.fusion_decoder(combined)
        # Text generation
        logits = self.text_decoder(fused)
        return logits

Key Results§

VLM-IE3D achieves state-of-the-art on ScanNetV2 for 3D detection (+5.2% mAP), 3D visual grounding (+4.8% accuracy), and 3D dense captioning (+3.1 CIDEr). On spatial reasoning benchmarks like 3D-CC, it outperforms prior VLMs by 6-8%.

Math Formulation§

The overall objective is a multi-task loss:

$$\mathcal{L} = \lambda_1 \mathcal{L}_{\text{det}} + \lambda_2 \mathcal{L}_{\text{ground}} + \lambda_3 \mathcal{L}_{\text{cap}} + \lambda_4 \mathcal{L}_{\text{reason}}$$

where each task uses standard losses (e.g., focal loss for detection, cross-entropy for captioning).