VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Technical Breakdown§

Overview§

VLK addresses the lack of paired egocentric images, language commands, and robot-compatible kinematic trajectories for humanoid loco-manipulation by generating synthetic supervision in reconstructed scenes. The pipeline uses 3D Gaussian Splatting (3DGS) to reconstruct metric-scale indoor environments, then synthesizes navigation and object-interaction trajectories using privileged scene information, and finally renders egocentric observations from the robot's perspective.

Core Methodology§

Scene Reconstruction

3D Gaussian Splatting represents a scene as a set of 3D Gaussians with mean $\mu$, covariance $\Sigma$, opacity $\alpha$, and color features. For each scene, a set of $N$ Gaussians $\mathcal{G} = \{g_i\}_{i=1}^N$ is optimized via differentiable rendering to match multi-view input images. The reconstruction provides metric-scale geometry and appearance, enabling physics-aware trajectory synthesis.

Trajectory Synthesis

Given the 3DGS scene, objects (e.g., a ball) are placed at random locations. Using privileged information (exact object positions, floor plan), collision-free navigation paths are generated via A* in a 2D occupancy grid. For object interaction, whole-body kinematic trajectories $\tau = (\mathbf{q}_t, \dot{\mathbf{q}}_t, \mathbf{p}_t^{\text{foot}}, \mathbf{p}_t^{\text{hand}})_{t=1}^T$ are computed by solving a quadratic program (QP) that minimizes joint torques while satisfying reachability, balance (ZMP constraint), and collision avoidance. The synthetic data tuple is $(\text{image}_t, \text{instruction}, \tau)$.

Policy Architecture

The VLK policy $\pi_\theta$ takes as input a sequence of $L$ past egocentric images $\{I_{t-L+1}, ..., I_t\}$, the language instruction $L$, and the robot's proprioceptive state $s_t$ (joint angles, angular velocities). Images are encoded by a frozen CLIP vision encoder, language by a frozen CLIP text encoder, and the embeddings are fused via cross-attention in a small transformer decoder. The policy outputs a short-horizon (e.g., 8 time-steps) sequence of whole-body pose deltas $\Delta \tau_t = (\Delta \mathbf{q}_{t+1}, ..., \Delta \mathbf{q}_{t+H})$. The training loss is a combination of mean squared error (MSE) on joint positions and velocities, plus a regularization term to avoid jitter.

import torch
import torch.nn as nn

class VLKPolicy(nn.Module):
    def __init__(self, img_dim=512, lang_dim=512, hidden_dim=256, horizon=8, num_joints=20):
        super().__init__()
        self.img_encoder = CLIPVisionEncoder(dim=img_dim, frozen=True)
        self.lang_encoder = CLIPTextEncoder(dim=lang_dim, frozen=True)
        self.cross_attn = nn.MultiheadAttention(embed_dim=hidden_dim, num_heads=4)
        self.transformer_decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model=hidden_dim, nhead=4), num_layers=2)
        self.output_proj = nn.Linear(hidden_dim, horizon * num_joints * 2)  # deltas for pos and vel

    def forward(self, imgs, lang, state):
        img_feats = self.img_encoder(imgs).mean(dim=1)  # (B, L', img_dim)
        lang_feat = self.lang_encoder(lang).unsqueeze(1)  # (B, 1, lang_dim)
        # fuse via cross-attention
        fused, _ = self.cross_attn(lang_feat, img_feats, img_feats)  # (B, 1, hidden_dim)
        # add state (proprioception) as a token
        state = state.unsqueeze(1)  # (B, 1, state_dim)
        combined = torch.cat([fused, state], dim=1)  # (B, 2, hidden_dim)
        # decode with zero query (start token)
        query = torch.zeros(combined.size(0), 1, hidden_dim).to(imgs.device)
        output = self.transformer_decoder(query, combined)  # (B, 1, hidden_dim)
        delta = self.output_proj(output.squeeze(1))  # (B, horizon*num_joints*2)
        return delta.view(-1, horizon, num_joints, 2)  # (B, H, J, 2)

Whole-Body Tracker

A separate whole-body tracker converts short-horizon kinematic predictions $\tau_{\text{pred}}$ into joint torques via inverse dynamics and a PD controller. The tracker runs at 1 kHz, while the policy runs at 50 Hz, ensuring smooth execution.

Training and Results§

Training uses 48,000 synthetic trajectories from 6 reconstructed indoor scenes. The policy is trained with AdamW (lr=3e-4) for 200 epochs. Evaluated on the Unitree G1 humanoid for navigation and object transport, the method achieves 85% success rate in sim and 70% in real (physical robot), demonstrating effective sim-to-real transfer without real-world data collection.