One Basis to Animate Them All: Gaussian Blendshape Distillation for Real-Time Avatars

Core Insight§

Pretrained 3D Gaussian avatars animate via a per-frame neural decoder that maps pose/expression codes $\mathbf{z}$ to deformed Gaussian parameters. This decoder is the runtime bottleneck. GALA shows the output of this decoder is approximately linear in identity-independent blendshape bases:

$$\mathbf{G}(\mathbf{z}, \mathbf{id}) \approx \mathbf{B}(\mathbf{id}) \cdot \mathbf{w}(\mathbf{z}) + \mathbf{b}(\mathbf{id})$$

where $\mathbf{B}(\mathbf{id})$ is a per-identity basis matrix, $\mathbf{w}(\mathbf{z})$ are the predicted blend coefficients, and $\mathbf{b}(\mathbf{id})$ is a static offset.

Distillation Pipeline§

1. Basis construction. Rather than a global PCA (which smears identity-specific detail), GALA builds block-local PCA over Gaussian attribute blocks (position, rotation, scale, opacity, SH coefficients), each under a rendering-aware metric and a fixed memory budget. This reduces per-identity memory while retaining high-frequency detail.

1. Coefficient predictor. A shallow MLP $f_\theta$ maps the animation code directly to blendshape weights:

$$\mathbf{w}(\mathbf{z}) = f_\theta(\mathbf{z})$$

This replaces the original heavy decoder entirely, and is trained by distillation against the frozen pretrained avatar's outputs.

1. Linear blend at inference. Final Gaussians are produced by one matrix multiply: $\mathbf{B}\mathbf{w} + \mathbf{b}$ — no neural decoding per frame.

import torch
import torch.nn as nn

class GALA(nn.Module):
    def __init__(self, code_dim, n_blendshapes, n_gaussians, attr_dim, hidden=128):
        super().__init__()
        # shallow coefficient predictor
        self.mlp = nn.Sequential(
            nn.Linear(code_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, n_blendshapes),
        )
        # per-identity basis B: [n_gaussians, attr_dim, n_blendshapes]
        self.register_buffer("B", torch.zeros(n_gaussians, attr_dim, n_blendshapes))
        self.register_buffer("b", torch.zeros(n_gaussians, attr_dim))

    def forward(self, z):
        w = self.mlp(z)                       # (N_b,)
        # linear blend -> (n_gaussians, attr_dim)
        gauss = torch.einsum("gab,b->ga", self.B, w) + self.b
        return gauss

# distillation loss against frozen teacher decoder
teacher.eval()
with torch.no_grad():
    target = teacher.decode(z, identity)
pred = student(z)
loss = ((pred - target) ** 2).mean()  # + rendering-aware basis loss
loss.backward()

Implementation & Results§

• Block-local PCA decomposes Gaussian attributes per spatial block, minimizing a rendering-aware reconstruction error subject to a memory cap.
• Identity generalization: the predictor is shared; only $\mathbf{B}, \mathbf{b}$ are per-identity, so it transfers to held-out identities.
• Model-agnostic: applied to three distinct avatar models (facial expressions, full-body with clothing dynamics) without retraining originals.
• Performance: up to three orders of magnitude CPU speedup, preserving most rendering quality, reaching 60fps on mobile devices.

The result empirically confirms that learned Gaussian avatar representations share a low-dimensional linear animation structure, enabling cheap, portable real-time animation.