FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations

FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations§

Core Problem and Contribution§

Modeling articulated objects (e.g., doors, drawers) from sparse monocular views is challenging due to partial geometry and motion evidence. Existing feed-forward methods rely on single-view category-level priors, limiting generalization. FAMOS is a feed-forward model that predicts movable-part segmentation and joint parameters from a sparse, unordered set of partial point clouds. It jointly reasons over multiple observations, supports variable input counts (including single view), and introduces two key innovations:

1. Multi-state Articulation Transformer (MAT) with alternating state-wise and global attention to aggregate articulation cues.
2. Observed Articulation Span (OAS) objective to supervise the motion range each part exhibits across observations.

Additionally, a procedural data generator synthesizes self-annotated assets during training to overcome dataset limitations.

Multi-state Articulation Transformer (MAT)§

Given a set of $N$ partial point clouds $\{P_i\}_{i=1}^N$ (each $P_i \in \mathbb{R}^{M \times 3}$), the model extracts per-point features via a shared PointNet-like encoder. These features are augmented with state embeddings (to distinguish observations) and then processed through $L$ transformer layers with alternating attention.

• State-wise attention: Within each observation, points attend to each other to capture local geometry.
• Global attention: Across all observations, points attend globally to fuse motion cues and infer articulation.

Formally, for layer $l$: $$H^{(l+1)} = \begin{cases} \text{Attention}(H^{(l)}) & \text{if } l \text{ odd (state-wise)} \\ \text{Attention}_{\text{global}}(H^{(l)}) & \text{if } l \text{ even (global)} \end{cases}$$ where state-wise attention masks out cross-observation connections. This alternating scheme allows the model to first refine per-view geometry, then integrate information across views, and repeat.

Observed Articulation Span (OAS) Objective§

To encourage the model to leverage all observations, OAS supervises the motion range each part exhibits. For each part $k$, let $\{\mathbf{T}_i^k\}_{i=1}^N$ be the predicted joint parameters (e.g., rotation angles, translation vectors) across observations. The observed span is defined as the range of these parameters: $$\text{span}^k = \max_i \|\mathbf{T}_i^k - \bar{\mathbf{T}}^k\| - \min_i \|\mathbf{T}_i^k - \bar{\mathbf{T}}^k\|$$ where $\bar{\mathbf{T}}^k$ is the mean. The loss $\mathcal{L}_{\text{OAS}} = \sum_k \|\text{span}^k - \text{span}_{\text{gt}}^k\|_1$ encourages the model to predict motion that matches the ground-truth span across the input observations. This is combined with segmentation loss $\mathcal{L}_{\text{seg}}$ (cross-entropy) and joint parameter loss $\mathcal{L}_{\text{joint}}$ (L1). Total loss: $\mathcal{L} = \lambda_1 \mathcal{L}_{\text{seg}} + \lambda_2 \mathcal{L}_{\text{joint}} + \lambda_3 \mathcal{L}_{\text{OAS}}$.

Procedural Data Generator§

To augment limited real datasets (PartNet-Mobility, ACD, ArtiCraft-10K), the authors design a procedural generator that synthesizes articulated objects with random geometries, joint types (revolute, prismatic), and motion ranges. It automatically produces ground-truth segmentation and joint parameters, enabling large-scale self-supervised training.

Implementation Details§

• Backbone: PointNet++ or similar point cloud encoder.
• Transformer: 6 layers, 8 heads, hidden dim 256.
• Training: AdamW optimizer, learning rate 1e-4, batch size 8, 100k iterations.
• Input: Variable number of point clouds, each with 512 points (sampled via farthest point sampling).

PyTorch Code Snippet: MAT Layer§

import torch
import torch.nn as nn

class MATLayer(nn.Module):
    def __init__(self, d_model, nhead, state_wise):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d_model, nhead, batch_first=True)
        self.norm1 = nn.LayerNorm(d_model)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, 4*d_model),
            nn.GELU(),
            nn.Linear(4*d_model, d_model)
        )
        self.norm2 = nn.LayerNorm(d_model)
        self.state_wise = state_wise

    def forward(self, x, state_ids):
        # x: (B, N*M, d_model), state_ids: (B, N*M) indicating observation index
        if self.state_wise:
            # Create attention mask: only attend within same state
            mask = (state_ids.unsqueeze(1) != state_ids.unsqueeze(2))  # (B, N*M, N*M)
            attn_out, _ = self.self_attn(x, x, x, attn_mask=mask)
        else:
            attn_out, _ = self.self_attn(x, x, x)
        x = x + attn_out
        x = self.norm1(x)
        x = x + self.ffn(x)
        x = self.norm2(x)
        return x

Results§

Experiments on PartNet-Mobility, ACD, and ArtiCraft-10K show consistent improvements over feed-forward and optimization-based baselines in part segmentation and joint parameter estimation. Ablations confirm the effectiveness of MAT and OAS.