ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation§

Overview ARDY is a streaming motion generation framework that produces high-fidelity 3D human motions in real-time, conditioned on online text prompts and kinematic constraints (e.g., keyframes, paths). It combines an autoregressive diffusion process with a hybrid motion representation to balance trajectory control and generative efficiency.

Hybrid Representation The motion at time step $t$ is decomposed into:

• Root features $\mathbf{r}_t$: explicit 2D/3D position, orientation, and velocity of the root joint (pelvis). These are kept in raw coordinate space to allow precise trajectory control.
• Latent body embedding $\mathbf{z}_t$: a compact latent representation of the remaining body joints, learned via a VAE-like encoder. This reduces dimensionality and enables efficient generative modeling.

The overall representation is $\mathbf{x}_t = [\mathbf{r}_t, \mathbf{z}_t]$, with the VAE decoder reconstructing full body pose from the latent.

Two-Stage Autoregressive Transformer Denoiser Diffusion is applied in a denoising autoregressive manner over the motion sequence. Let $\mathbf{x}^{k}_t$ denote the noisy representation at denoising step $k$ for time step $t$. The process consists of: 1. Stage 1 (Latent denoising): A transformer models the conditional distribution $p(\mathbf{x}^{k}_t | \mathbf{x}^{k}_{<t}, \mathbf{c})$, where $\mathbf{c}$ includes text condition, history context (past clean motions), and kinematic constraints. It predicts the denoised latent $\mathbf{z}_t$ given root features. 2. Stage 2 (Root refinement): A separate small network refines root features $\mathbf{r}_t$ conditioned on the denoised latent and constraints, ensuring trajectory adherence.

Both stages use a transformer with variable history context length and causal masking.

Training The VAE is first trained on motion clips. Then the diffusion model is trained to predict clean $\mathbf{x}_t$ from noisy versions, with condition masking (randomly dropping text or constraints) to enable flexible online control. The loss is the mean squared error between predicted and ground truth clean motion.

Inference At test time, autoregressive denoising generates motion frame-by-frame. Text prompts can be changed arbitrarily, and kinematic constraints (e.g., keyframe poses, path waypoints) are injected as conditioning. The hybrid representation allows the root to precisely follow constraints while the body flows naturally.

Code Snippet (PyTorch-like illustration)

class HybridDenoiser(nn.Module):
    def __init__(self, d_root, d_latent, d_model, n_layers):
        super().__init__()
        self.vae = PretrainedVAE()  # encoder/decoder for latent body
        self.root_encoder = nn.Linear(d_root, d_model)
        self.latent_encoder = nn.Linear(d_latent, d_model)
        self.transformer = TransformerDecoder(d_model, n_layers)
        self.latent_head = nn.Linear(d_model, d_latent)
        self.root_head = nn.Linear(d_model, d_root)

    def forward(self, noisy_root, noisy_latent, cond, mask):
        # noisy_root shape: (B, T, d_root), noisy_latent: (B, T, d_latent)
        r = self.root_encoder(noisy_root)
        z = self.latent_encoder(noisy_latent)
        x = r + z  # sum or concatenate
        x = self.transformer(x, context=cond, mask=mask)
        pred_latent = self.latent_head(x)
        pred_root = self.root_head(x)
        return pred_root, pred_latent

Key Equations

• Diffusion objective:

$$\mathcal{L} = \mathbb{E}_{t,k,\epsilon} \left[ \| \mathbf{x}_t - \hat{\mathbf{x}}_\theta(\mathbf{x}_t^k, \mathbf{c}, k) \|^2 \right]$$

• Autoregressive factorization:

$$p(\mathbf{x}_{1:T}) = \prod_{t=1}^T p(\mathbf{x}_t | \mathbf{x}_{<t})$$

Results ARDY achieves state-of-the-art quality on HumanML3D and the Bones Rigplay dataset while enabling real-time inference (<30ms per frame). The hybrid representation is crucial for balancing trajectory fidelity (root) and natural body motion (latent).