4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes§

Core Problem: Inverse graphics aims to recover scene structure and dynamics from visual observations. While static 3D reconstruction has advanced, reconstructing dynamic scenes (4D) with physical realism remains challenging. This paper introduces a benchmark to evaluate LLM agents' ability to generate code that simulates dynamic scenes from video.

Methodology: The benchmark provides an agent with a video of a dynamic scene (e.g., deformation, fluid flow, fracture) and requires it to output an executable graphics program (e.g., using a physics engine like Taichi or PyBullet) that reproduces the observed dynamics. The agent must infer latent physical parameters, object geometries, and simulation rules purely from pixels. The evaluation compares rendered simulations against ground-truth videos using perceptual metrics (e.g., LPIPS, FVD) and physical plausibility checks.

Implementation Details:

• Dataset: Real-world videos and synthetic scenes covering diverse phenomena: elastic deformation, fluid flow, fracture, rigid body dynamics.
• Agent Interface: The agent receives a video and a code scaffold (e.g., a Python API for physics simulation). It iteratively generates code, which is executed, rendered, and compared to the input video.
• Evaluation: Metrics include visual similarity (PSNR, SSIM, LPIPS), dynamics alignment (e.g., optical flow consistency), and physical parameter error (if ground-truth parameters are available).
• Frontier Models: Extensive benchmarking of LLMs (e.g., GPT-4, Claude) shows that strong static reconstruction does not guarantee dynamic reconstruction; agents struggle with complex phenomena like fluids and fracture.

Code Snippet: The following illustrates a simplified agent loop for inverse graphics via code generation using a physics engine:

import torch
import taichi as ti

# Agent generates simulation code (string) given video frames
def generate_simulation_code(video_frames):
    # Use LLM to generate code based on video
    code = llm.generate(prompt=f"Generate Taichi code to simulate: {video_frames}")
    return code

# Execute generated code and render
def render_simulation(code, num_steps=100):
    exec(code, globals())
    frames = []
    for step in range(num_steps):
        simulate_step()  # defined in generated code
        frames.append(render())  # render current state
    return torch.stack(frames)

# Evaluate against ground truth video
def evaluate(rendered, gt_video):
    lpips_loss = lpips(rendered, gt_video)
    flow_consistency = compute_flow_consistency(rendered, gt_video)
    return lpips_loss, flow_consistency

Key Findings:

• LLM agents achieve moderate success on simple rigid body dynamics but fail on complex phenomena.
• Performance correlates with the complexity of the physical simulation required.
• The benchmark provides a testbed for tracking progress toward agents that can interpret world dynamics through code.

Mathematical Formulation: The inverse graphics problem can be framed as finding a program $P$ that minimizes the difference between rendered output and observed video:

$$\min_{P \in \mathcal{P}} \mathcal{L}(\text{Render}(P), V)$$

where $\mathcal{P}$ is the space of executable programs, $V$ is the input video, and $\mathcal{L}$ is a perceptual loss. The agent must search this program space using natural language and code generation capabilities.

Conclusion: 4DCodeBench highlights the gap between static and dynamic scene understanding in LLM agents and provides a challenging benchmark for future research.