SM4RT: Learning Structured Motion Geometry for 4D Reconstruction

Overview§

SM4RT proposes a novel representation called Structure-of-Motion, where scene dynamics are decomposed into a set of motion bases, each a temporal sequence of 6D twists in SE(3). Dense motion is recovered via sparse, time-shared per-pixel assignment weights, enforcing rigid-body kinematics.

Core Method§

Structure-of-Motion Representation§

Scene motion is modeled as a combination of $K$ rigid-body transformations. Each motion base $\mathcal{M}_k$ is a sequence of twists $\boldsymbol{\xi}_{k,t} \in \mathbb{R}^6$ over time $t=1...T$. The twist parameterizes a rigid transformation $\mathbf{T}_{k,t} = \exp(\boldsymbol{\xi}_{k,t}^\wedge) \in SE(3)$. Dense motion field $\mathbf{M}_t \in \mathbb{R}^{H \times W \times 3}$ is computed as:

$$\mathbf{M}_t = \sum_{k=1}^K \mathbf{W}_k \odot \left( \mathbf{T}_{k,t} \circ \mathbf{X} \right)$$

where $\mathbf{W}_k \in \mathbb{R}^{H \times W}$ are assignment weights (sparse, time-shared), $\mathbf{X} \in \mathbb{R}^{H \times W \times 3}$ is the canonical 3D geometry, and $\circ$ applies the transformation to each 3D point.

Architecture§

SM4RT uses a parallel motion geometry encoder and decoder. The encoder extracts features from monocular RGB frames. The decoder jointly predicts:

• Canonical 3D geometry $\mathbf{X}$ (a point map)
• Motion bases $\{\boldsymbol{\xi}_{k,t}\}$
• Assignment weights $\{\mathbf{W}_k\}$

Training is end-to-end with losses on 3D geometry, motion flow, and a sparsity prior on weights.

Code Snippet§

import torch
import torch.nn as nn

class MotionBasePredictor(nn.Module):
    def __init__(self, K=10, T=32):
        super().__init__()
        self.K = K
        self.T = T
        self.twist_mlp = nn.Linear(512, 6*T*K)  # predicts all twists

    def forward(self, features):
        # features: (B, 512, H, W)
        B, _, H, W = features.shape
        twist_params = self.twist_mlp(features.mean(dim=[2,3]))  # (B, 6*T*K)
        twist_params = twist_params.view(B, self.K, self.T, 6)
        return twist_params

class WeightPredictor(nn.Module):
    def __init__(self, K=10):
        super().__init__()
        self.conv = nn.Conv2d(512, K, kernel_size=1)

    def forward(self, features):
        logits = self.conv(features)  # (B, K, H, W)
        weights = torch.softmax(logits, dim=1)  # sparse via L1
        return weights

Key Contributions§

• Novel Structure-of-Motion representation that enforces rigid-body kinematics.
• End-to-end 4D reconstruction from monocular video without explicit tracking.
• State-of-the-art performance on dynamic scene reconstruction benchmarks.

Results§

SM4RT achieves superior motion reconstruction while preserving geometric structure, outperforming prior flow-based and tracking-based methods on datasets like DAVIS and KITTI.