Pixel: A Co-Designed Full-Stack Robot System for Learning General-Purpose Mobile Manipulation

Pixel: A Co-Designed Full-Stack Robot System§

Pixel is an integrated system for general-purpose mobile manipulation, combining hardware, data collection, control, and learning in a co-designed manner. The goal is to address mismatches that arise when components are developed independently.

Hardware§

• Low-cost wheeled humanoid: Designed with human-aligned kinematics and compliant grippers to facilitate teleoperation and human-like manipulation.
• Compliant grippers: Enable safe interaction and robust grasping.

Data Collection§

• Multiple teleoperation interfaces: Including robot-free human data collection (e.g., using VR or motion capture) to scale demonstration gathering.
• Common policy learning pipeline: Learns from both human and robot demonstrations, bridging the embodiment gap.

Control§

• Unified whole-body controller: Coordinates wheeled base and arms for mobile manipulation tasks. The controller likely uses inverse kinematics and dynamic control to achieve desired end-effector poses while maintaining balance.

Learning§

• Hierarchical learning stack: Enables language-conditioned visuomotor control and long-horizon task execution.
• High-level policy: Maps language instructions and observations to subgoals or primitive actions.
• Low-level policy: Executes visuomotor control to achieve subgoals.
• Policy learning pipeline: Trained via imitation learning (behavior cloning) on demonstrations. Possibly uses diffusion policies or transformers for multimodal action distributions.

Methodology Details§

• Imitation Learning: The policy $\pi_\theta(a_t | o_t, g)$ is trained to minimize the negative log-likelihood of expert actions: $$\mathcal{L}(\theta) = -\mathbb{E}_{(o_t, a_t, g) \sim \mathcal{D}} [\log \pi_\theta(a_t | o_t, g)]$$
• Hierarchical Control: A high-level planner generates subgoals $g_{1:T}$ from language instruction $l$ and observations, and a low-level controller executes actions to reach each subgoal.
• Whole-Body Control: The controller computes joint torques or velocities to track desired end-effector poses while satisfying constraints. For a wheeled base, it might use a differential drive model and solve a quadratic program (QP) for whole-body control.

Implementation Snippet§

Below is a simplified PyTorch snippet illustrating a hierarchical policy training loop with behavior cloning.

import torch
import torch.nn as nn
import torch.optim as optim

class HighLevelPolicy(nn.Module):
    def __init__(self, lang_dim, obs_dim, subgoal_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(lang_dim + obs_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 256),
            nn.ReLU(),
            nn.Linear(256, subgoal_dim)
        )
    def forward(self, lang_emb, obs):
        return self.net(torch.cat([lang_emb, obs], dim=-1))

class LowLevelPolicy(nn.Module):
    def __init__(self, obs_dim, subgoal_dim, action_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim + subgoal_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 256),
            nn.ReLU(),
            nn.Linear(256, action_dim)
        )
    def forward(self, obs, subgoal):
        return self.net(torch.cat([obs, subgoal], dim=-1))

# Training loop
high_policy = HighLevelPolicy(lang_dim=64, obs_dim=128, subgoal_dim=10)
low_policy = LowLevelPolicy(obs_dim=128, subgoal_dim=10, action_dim=7)
optimizer = optim.Adam(list(high_policy.parameters()) + list(low_policy.parameters()), lr=1e-3)

for epoch in range(num_epochs):
    for lang_emb, obs, subgoal, action in dataloader:
        # High-level: predict subgoal from language and observation
        pred_subgoal = high_policy(lang_emb, obs)
        # Low-level: predict action from observation and subgoal
        pred_action = low_policy(obs, pred_subgoal)
        # Loss: MSE for subgoal and action (or negative log-likelihood for stochastic policies)
        loss = nn.MSELoss()(pred_subgoal, subgoal) + nn.MSELoss()(pred_action, action)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Evaluation§

• Demonstrated mobile-manipulation behaviors requiring dexterity, active perception, navigation, and whole-body coordination.
• Evaluated across hardware, data collection, and policy learning, showing the benefits of co-design.

Pixel provides a foundation for studying how robot hardware, data, control, and learning can be designed together to advance robot learning.