GraphVid: Interactive Graph-Controllable Video Generation

Technical Synopsis§

Core Methodology§

GraphVid is an image-to-video generation model conditioned on a structured interaction graph $G = (V, E)$, where nodes $V$ represent objects (with initial bounding boxes and classes) and edges $E$ represent interactions (e.g., 'push', 'carry', 'follow'). The graph provides a compact, compositional representation of multi-object dynamics, allowing precise specification of interactions without requiring accurate trajectory drawings.

The model builds on a latent diffusion backbone (similar to Stable Video Diffusion) and injects graph information via cross-attention layers. The graph is encoded using a graph neural network (GNN) that produces per-object embeddings, which are then aggregated with spatial positional encodings. The conditioning mechanism combines the initial image latent $z_0$, noise $\epsilon_t$, timestep $t$, and graph embedding $g$ to predict the noise at step $t$:

$$\hat{\epsilon}_\theta(z_t, t, z_0, g) = \epsilon_\theta(z_t, t, \text{concat}(z_0, \phi(g)))$$

where $\phi$ is a learned projection. The training objective is the standard denoising MSE:

$$\mathcal{L} = \mathbb{E}_{z_0, \epsilon, t, g} \left[ \| \epsilon - \hat{\epsilon}_\theta(z_t, t, z_0, g) \|^2 \right]$$

Dataset: GraphVid-Bench§

To support graph-conditioned training, the authors curate GraphVid-Bench, a large-scale dataset of 10,000+ videos with interaction graphs. Videos are sourced from existing datasets (e.g., Something-Something, Epic-Kitchens) and annotated with object tracks and interaction labels using a semi-automated pipeline (object detector + tracker + relationship classifier). Each video has ground-truth graphs at each frame, providing dense supervision.

Implementation Details§

The model uses a U-Net architecture with spatio-temporal attention blocks. The GNN encoder is a 2-layer Graph Isomorphism Network (GIN) with 256 hidden dimensions. Cross-attention is inserted into each U-Net block after the temporal attention layer. Training uses 8×A100 GPUs, batch size 64, learning rate 1e-4, and 200k steps. The model has 1.4B parameters, significantly fewer than Motion-I2V (3.5B).

import torch
import torch.nn as nn
from torch_geometric.nn import GINConv

class GraphEncoder(nn.Module):
    def __init__(self, hidden_dim=256):
        super().__init__()
        self.conv1 = GINConv(nn.Sequential(nn.Linear(128, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim)))
        self.conv2 = GINConv(nn.Sequential(nn.Linear(hidden_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim)))
        self.proj = nn.Linear(hidden_dim, 1024)  # match U-Net conditioning dimension

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index).relu()
        return self.proj(x)

# Inside U-Net forward pass:
# g_emb = graph_encoder(node_features, edge_index)  # [B, N, 1024]
# g_emb = g_emb.mean(dim=1)  # global graph embedding [B, 1024]
# Then injected via cross-attention with latent tokens

Results and Comparison§

Compared to trajectory-based Motion-I2V, GraphVid achieves 39.9% lower FID (13.2 vs 21.9), 37.6% lower FVD (112.4 vs 180.1), PSNR improvement (9.87 → 15.98), and SSIM improvement (0.38 → 0.61). User studies show GraphVid is preferred for interaction fidelity and temporal consistency. The graph interface also enables interactive editing (add/remove objects, modify edges) with minimal re-generation.

Key Takeaways§

• Graph-based conditioning provides a structured, scalable alternative to pixel-level control for multi-object video generation.
• The dataset and model demonstrate that interaction graphs can be learned from sparse annotations and generalize to complex scenes.
• The approach reduces data and parameter requirements while improving quality, suggesting structured semantic interfaces as a promising paradigm.