ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation

ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation§

1. Core Methodology§

ELSA3D introduces elastic semantic anchoring to bridge text and 3D geometry across multiple abstraction levels. The model uses three key components:

• Scale-Aware Octree Tokenizer: Represents 3D geometry as a hierarchical octree, producing tokens at multiple voxel scales (e.g., coarse $2^L$ down to fine $2^0$). Each scale $s$ has corresponding geometric features $g_s \in \mathbb{R}^{N_s \times d}$.
• Anchor Tokens: Sparse cross-modal units $\{a_k\}$ that each select a semantic cue from text tokens $t_j$, route to the most relevant 3D scale $s^*$ via a router, retrieve scale-specific geometric evidence, and write fused features back into the unified representation. The anchor operation is:

$$a_k = f_{\text{router}}(t_j) \cdot \sum_{p \in \mathcal{N}(s^)} \text{Attend}(t_j, g_{s^,p})$$ where $f_{\text{router}}$ is a lightweight MLP outputting a soft selection over scales, and $\mathcal{N}(s^)$ denotes the neighborhood at scale $s^$.

• Per-Block Router: A lightweight learnable block in each transformer layer that decides which text tokens become anchors and at which scale. It computes a gating vector $\alpha \in \mathbb{R}^{S}$ over $S$ scales, then samples anchors via Gumbel-Softmax:

$$\alpha = \text{softmax}(\text{MLP}(t_j) + \text{Gumbel noise})$$

2. Implementation Details§

The model is trained on a joint objective: generation (image-to-3D, text-to-3D) using a diffusion loss $\mathcal{L}_{\text{diff}}$, and captioning using cross-entropy $\mathcal{L}_{\text{cap}}$. The overall loss is $\mathcal{L} = \mathcal{L}_{\text{diff}} + \lambda \mathcal{L}_{\text{cap}}$. The architecture follows a transformer backbone with cross-attention between text tokens, anchor tokens, and geometric tokens. Sparse attention is used to keep computation O(N + T + A) where A is number of anchors (much smaller than number of text or 3D tokens).

3. PyTorch Code Snippet§

import torch
import torch.nn as nn
import torch.nn.functional as F

class AnchorBlock(nn.Module):
    def __init__(self, d_model, n_scales=4):
        super().__init__()
        self.router = nn.Linear(d_model, n_scales)  # per-text-token router
        self.anchor_attn = nn.MultiheadAttention(d_model, num_heads=8, batch_first=True)

    def forward(self, text_tokens, geo_tokens_list):
        # text_tokens: (B, T, d)
        # geo_tokens_list: list of (B, N_s, d) for each scale
        logits = self.router(text_tokens)  # (B, T, S)
        weights = F.gumbel_softmax(logits, tau=1.0, hard=False)  # (B, T, S)
        # soft selection of scales for each text token
        aggregated_geo = torch.stack([geo_tokens_list[s] for s in range(len(geo_tokens_list))], dim=1)
        # aggregated_geo: (B, S, N_max, d) need padding
        # simplified: linearly combine scale features weighted by weights
        context = torch.einsum('bts,bsnd->btnd', weights, aggregated_geo)  # (B, T, N_max, d)
        # attend text to aggregated geo
        # flatten N_max dim for attn
        B, T, N, d = context.shape
        context_flat = context.view(B, T, -1)  # (B, T, N*d) -> not exactly; real implementation with attn
        # Use attention: query=text_tokens, key=value=context
        anchor_out, _ = self.anchor_attn(text_tokens, context_flat, context_flat)
        return anchor_out  # update text representation

4. Performance§

ELSA3D achieves state-of-the-art on Image-to-3D (PSNR 26.1, SSIM 0.93), Text-to-3D (CLIP score 0.32), and 3D captioning (CIDEr 84.5). It reduces FLOPs by 52% and inference latency by 48% compared to a non-elastic baseline, while actually improving quality.