HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark

Overview§

HumanTracker addresses evaluation of humanoid motion tracking. Traditional kinematic metrics such as MPJPE average per-frame joint errors but fail to penalize perceptual artifacts like foot skating, unstable support, and mistimed contacts. The benchmark provides 153 hours of optical motion data from professional performers across four motion families, plus text labels for fine-grained diagnostics.

HumanScore§

HumanScore is a learned, preference-aligned metric. It is trained on 12K paired comparisons comprising 24K motions. Rather than hand-crafting contact features, it learns to score a tracked motion in a way that agrees with human judgments. Given two tracked motions $x_i$ and $x_j$, the model predicts preference probability using a Bradley-Terry model:

$$P(x_i \succ x_j) = \sigma(f_\theta(x_i) - f_\theta(x_j))$$

The scalar score $f_\theta(x)=w^T h$ is computed by a temporal encoder over pose/contact features. Training minimizes binary cross-entropy on preference labels $y_{ij} \in \{0,1\}$.

Implementation§

The encoder is a lightweight transformer operating on sequences of per-frame pose and contact features (joint positions, velocities, foot contact one-hots). Masked mean-pooling aggregates the temporal dimension, followed by a linear head.

import torch
import torch.nn as nn
import torch.nn.functional as F

class HumanScore(nn.Module):
    def __init__(self, d_input=96, d_model=128, nhead=4, layers=3):
        super().__init__()
        self.proj = nn.Linear(d_input, d_model)
        self.encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model, nhead, batch_first=True),
            num_layers=layers
        )
        self.head = nn.Linear(d_model, 1)

    def forward(self, x, mask=None):
        x = self.proj(x)                     # (B, T, d_model)
        h = self.encoder(x, src_key_padding_mask=mask)
        h = h.masked_fill(mask.unsqueeze(-1), 0).sum(1) / (~mask).sum(1, keepdim=True)
        return self.head(h).squeeze(-1)

model = HumanScore()
optim = torch.optim.Adam(model.parameters(), lr=1e-4)

# Pairwise training loop
for xi, xj, pref in loader:
    si, sj = model(xi), model(xj)
    loss = F.binary_cross_entropy_with_logits(si - sj, pref)
    optim.zero_grad(); loss.backward(); optim.step()

Benchmark Design§

The 153-hour dataset uses optical motion capture, avoiding monocular noise and providing accurate contact labels. Four motion families cover locomotion, manipulation, transitions, and acrobatics, ensuring contact-rich and long-horizon behaviors. Text labels allow per-skill failure diagnosis. Evaluation shows that HumanScore aligns better with human preference than existing pose metrics and detects stability/contact errors that kinematic metrics miss, making it a scalable and perceptually valid training and evaluation signal for humanoid teleoperation and whole-body imitation.