OpenCoF: Learning to Reason Through Video Generation

Core Methodology§

OpenCoF introduces Chain-of-Frame (CoF) reasoning, where a video generation model reasons by producing temporally connected frames. The framework consists of:

• OpenCoF-17K Dataset: 17K reasoning videos spanning 11 task families (e.g., causal, spatial, analogical). Each video is a sequence of frames that illustrate a reasoning chain.
• Wan-CoF Model: Fine-tuned from Wan2.2-I2V-A14B, a text-to-video diffusion model. The finetuning uses the OpenCoF-17K dataset with a simple denoising objective:

$$\mathcal{L}_{\text{simple}} = \mathbb{E}_{\mathbf{x}_0, \epsilon, t} \left[ \| \epsilon - \epsilon_\theta(\mathbf{x}_t, t, \mathbf{c}) \|^2 \right]$$

where $\mathbf{c}$ is the conditioning (text prompt + initial frame).

Advanced Design: Reasoning Tokens§

To enhance CoF capabilities, the authors equip the model with two types of learnable tokens:

• Visual Reasoning Tokens ($\mathbf{r}_v$) : Capture low-level visual cues for spatial reasoning (e.g., object positions, motion).
• Textual Reasoning Tokens ($\mathbf{r}_t$) : Capture high-level semantic priors for temporal reasoning (e.g., causal relations).

These tokens are prepended to the frame token sequence before the transformer layers. The attention mechanism allows them to interact with all frames, acting as a scratchpad:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

where $Q$, $K$, $V$ include both frame tokens and reasoning tokens.

Implementation Details§

• Architecture: Wan2.2-I2V-A14B is a large video diffusion model with ~14B parameters, using a 3D VAE and a text encoder.
• Training: Fine-tuned on OpenCoF-17K for 10K steps, with batch size 64, learning rate 1e-4, and mixed precision.
• Inference: Use DDIM sampling with 50 steps, conditioning on text prompt and first frame.

Code Illustration§

import torch
import torch.nn as nn

class ReasoningCoFModel(nn.Module):
    def __init__(self, base_model, num_visual_tokens=8, num_textual_tokens=8, token_dim=4096):
        super().__init__()
        self.base_model = base_model  # Wan2.2 backbone
        self.visual_reasoning_tokens = nn.Parameter(torch.randn(1, num_visual_tokens, token_dim))
        self.textual_reasoning_tokens = nn.Parameter(torch.randn(1, num_textual_tokens, token_dim))
        
    def forward(self, noisy_frames, timesteps, text_embedding):
        batch_size = noisy_frames.shape[0]
        v_tokens = self.visual_reasoning_tokens.expand(batch_size, -1, -1)
        t_tokens = self.textual_reasoning_tokens.expand(batch_size, -1, -1)
        # Concatenate tokens with frame token sequence
        tokens = torch.cat([v_tokens, t_tokens, noisy_frames], dim=1)
        # Pass through transformer blocks (simplified)
        output = self.base_model.transformer(tokens, timesteps, text_embedding)
        # Remove reasoning tokens from output
        frame_output = output[:, -noisy_frames.shape[1]:, :]
        return frame_output

Key Results§

• On four video reasoning benchmarks (Causal, Spatial, Analogical, Temporal), Wan-CoF achieves ~15-20% relative improvement over the Wan2.2 baseline.
• Attention analysis reveals that visual tokens focus on spatial regions, while textual tokens attend more uniformly across time, supporting their complementary roles.

Conclusion§

The paper demonstrates that explicit reasoning tokens and diverse temporal supervision are both necessary for strong video-based reasoning, paving the way for reasoning-oriented video generation.