The Seriality Gap in Video Diffusion Models

Overview§

The paper identifies a fundamental limitation of standard bidirectional video diffusion models: they perform poorly on tasks requiring sequential causal reasoning. Using multi-ball hard-sphere dynamics as a controlled testbed, the authors show that performance degrades with longer causal chains, even with more denoising steps. This degradation is not due to video length per se (single-ball control shows minimal degradation) but to the requirement for serial computation over dependent events.

Methodology§

Experiments use a physics simulation of 2D bouncing balls. Input: initial positions/velocities; output: future frames. Key conditions:

• Multi-ball: balls interact via collisions, creating causal chains (e.g., ball A hits B, then B hits C).
• Single-ball: no interactions, same video length control.

Models: standard bidirectional video diffusion (e.g., from Ho et al.). The denoising process is a reverse Markov chain with $T$ steps, each using a shared U-Net backbone. The authors measure pixel MSE and perceptual metrics.

Key Results§

• Performance on multi-ball tasks degrades as the number of balls (causal chain length) increases, while single-ball performance stays flat.
• Increasing denoising steps (e.g., from 100 to 1000) does not help multi-ball performance, confirming that serial compute does not scale with $T$.
• Interventions that increase effective serial computation improve results:
• Autoregressive/blockwise generation (generate frames sequentially, conditioning on previous outputs).
• Deeper backbones (more layers per denoising step).

Theoretical Analysis§

The authors prove that for deterministic video prediction, the denoising process is equivalent to a single function evaluation of the backbone plus a linear mapping. Specifically, the reverse step $x_{t-1} = f(x_t, t)$ can be reparameterized so that the final prediction $x_0$ is a deterministic function of $x_T$ and the backbone, with no extra serial computation across $t$. Formally:

$$x_0 = \text{Backbone}(x_T, t=0)$$

Thus, adding denoising steps does not increase the serial depth of the computation; only architectural depth (number of backbone layers) or temporal autoregression can provide the needed serial reasoning.

Implementation Code Snippet§

Below is a simplified PyTorch-like illustration of the diffusion forward and reverse processes highlighting the lack of serial compute scaling.

import torch
import torch.nn as nn

class SimpleDiffusion(nn.Module):
    def __init__(self, backbone):
        super().__init__()
        self.backbone = backbone  # e.g., U-Net

    def forward(self, x_T, T):
        # Standard reverse: single-step denoising (deterministic)
        x = x_T
        for t in reversed(range(T)):
            # Each step uses same backbone with time embedding
            x = self.backbone(x, t)
        return x

However, the paper shows that this loop does not add seriality: the final output is equivalent to a single backbone call. To add serial computation, use autoregressive generation:

def autoreg_generate(model, init_frames, num_frames):
    frames = []
    context = init_frames
    for _ in range(num_frames):
        next_frame = model.denoise(context)  # conditioned on past
        frames.append(next_frame)
        context = torch.cat([context, next_frame], dim=1)
    return frames

Conclusions§

Practitioners should avoid relying solely on increasing denoising steps for tasks requiring causal reasoning. Instead, adopt autoregressive decoding or deeper architectures to provide the necessary serial computation. The "seriality gap" is a structural limitation of current video diffusion models, opening avenues for new architectures that interleave diffusion with explicit sequential processing.