AVA-Encoder: Towards Agent-Native Video Representation Learning

Overview§

AVA-Encoder is an agentic auto-encoder that compresses video into a structured knowledge graph (KG) and reconstructs it. The KG comprises hierarchy/state nodes storing structured text and a linked asset layer containing generated images, audio, and video. Typed edges capture relations in a form that agents can query, reason over, and edit.

Methodology§

The core idea is to make the latent representation itself agent-native. Given a video $V$, the encoder $E_\theta$ produces a graph $G = E_\theta(V)$. The decoder $D_\phi$ reconstructs $\hat{V} = D_\phi(G)$. Standard auto-encoding would optimize: $$\mathcal{L}_{\text{recon}} = \| V - D_\phi(E_\theta(V)) \|_1.$$ However, pixel-space gradients do not align with agent reasoning. AVA-Encoder instead uses a textual-gradient optimization framework. Reconstruction differences are evaluated and expressed as natural-language update directions $\Delta_{\text{txt}}$. In the outer loop, these textual gradients drive Data-Independent Encoding Policy Pseudo-Training: the agentic policy $\pi$ learns to propose KG modifications that reduce reconstruction error. The inner loop optionally performs Data-Dependent KG Representation Refinement at test time, editing the graph $G$ with policy-suggested changes before decoding.

The overall training objective can be written as: $$\min_{\theta, \phi, \pi} \mathbb{E}_V \left[ \mathcal{L}_{\text{recon}}(V, D_\phi(G)) + \lambda \mathcal{L}_{\text{agent}}(G, \Delta_{\text{txt}}; \pi) \right],$$ where $\mathcal{L}_{\text{agent}}$ measures how well the KG supports subsequent agentic queries and edits.

Implementation§

The KG is realized as a multimodal graph: state nodes contain textual descriptions, asset nodes link to audio/video/image tensors, and typed edges such as causes, follows, and subject define temporal and causal structure. The decoder is a conditional video generator conditioned on serialized graph triples. The policy is an LLM prompted to produce textual diffs, e.g., "increase tension by shortening the shot duration and adding a close-up."

The code below sketches the training loop:

class AVAEncoder(nn.Module):
    def __init__(self, kg_encoder, decoder, policy_llm):
        super().__init__()
        self.kg_encoder = kg_encoder   # video -> KG
        self.decoder = decoder         # KG -> video
        self.policy_llm = policy_llm   # textual-gradient policy

    def forward(self, video):
        kg = self.kg_encoder(video)
        recon = self.decoder(kg)
        return kg, recon

for video in dataloader:
    kg, recon = model(video)
    loss_recon = F.mse_loss(recon, video)

    feedback = evaluator(video, recon)          # text critique
    update_txt = model.policy_llm(feedback, kg) # textual gradient
    loss_policy = policy_loss(kg, update_txt)

    (loss_recon + 0.1 * loss_policy).backward()
    optimizer.step()

Results§

AVA-Encoder outperforms the strongest external baseline by 20.7 percentage points on a reliable agentic video reconstruction benchmark. In a controlled policy-only setting, its pseudo-trained shot-level Agentic Video Encoder policy also beats a carefully human-tuned policy while using 74.3% fewer system-prompt tokens. The authors release the complete framework, the benchmark, and the first dataset of high-quality film KG representations.