The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

Overview§

This paper proposes a diagnostic evaluation framework for video language models (VLMs) on event counting tasks, moving beyond aggregate accuracy to timestamp-level auditing against executable ground truth. The authors generate 2,190 controlled videos across three tasks (bouncing-ball wall contacts, visual blinks, and categorical state transitions), varying event count $N$ and frequency $F$ while keeping rendering fixed. Each video is paired with an executable event trace, enabling capability-surface estimation and failure-mode isolation.

Methodology§

Parametric Video Generation§

Videos are synthetically rendered with precise control over event count and frequency. For each combination of $(N, F)$, rendering parameters remain constant, ensuring that performance differences are attributable to temporal reasoning rather than visual complexity. Three task types probe distinct event representations:

• Persistent state transitions (e.g., color changes) remain visible after the event.
• Transient events (e.g., blinks) disappear quickly.
• Categorical switches between discrete states.

Trace-Grounded Evaluation§

Instead of scoring only the final count, the authors compare model-generated event sequences against the ground-truth trace. They compute several metrics:

• Final count accuracy: exact match of reported count $\hat{N}$ to true count $N$.
• Event recovery rate: the fraction of true events correctly identified, $\text{Rec} = \frac{|\text{TP}|}{N}$.
• Sequence agreement: exact temporal alignment of the reported event timestamps $\{\hat{t}_i\}$ with ground-truth $\{t_i\}$ within a tolerance.

Reliability is defined as achieving at least an 80% success rate at a given $(N, F)$ operating point.

Key Findings§

The results reveal a staged temporal failure:

• Gemini 3.6 Flash reliably counts persistent state transitions up to 12 events at 0.5–1.0 Hz, but shows no reliable positive-count region for transient blinking events.
• At high count and high frequency, only 0.2% of final counts are correct, and event recovery drops to 18.1%.
• Increasing the sampling rate (frames per second) improves Bounce Ball accuracy from 19.6% to 29.3%, but sequence agreement remains at 3.7%. This indicates that extra frames inflate final scores without producing faithful event recovery.
• Prompting strategies provide limited gains, and real-world video evaluations show the same concentration of success at low event counts.

The authors argue that the event representation dictates whether a model can initially access evidence, and this limitation compounds as count and frequency increase.

Code Illustration§

The following pseudo-PyTorch snippet illustrates how one might evaluate a VLM's event sequence against a ground-truth trace using timestamp-level matching:

import torch

def sequence_agreement(pred_times, true_times, tolerance=0.5):
    """Compute exact-alignment ratio within tolerance."""
    pred_t = torch.tensor(pred_times)
    true_t = torch.tensor(true_times)
    matches = 0
    for t in true_t:
        if torch.any(torch.abs(pred_t - t) <= tolerance):
            matches += 1
    return matches / len(true_t)

# Example: model outputs timestamps for detected events
model_times = [0.8, 2.1, 3.9, 6.0]
gt_times = [1.0, 2.0, 4.0, 5.0]
print(sequence_agreement(model_times, gt_times))  # 0.75

Equations§

Reliability threshold at operating point $(N, F)$: $$ \text{Rel}(N,F) = \frac{1}{K} \sum_{k=1}^{K} \mathbb{I}[\text{Acc}_k \geq 0.8] $$

Event recovery rate: $$ \text{Rec} = \frac{\text{TP}}{\text{TP} + \text{FN}} $$

Trace-grounded accuracy: $$ \text{Acc}_{\text{trace}} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}[|\hat{t}_i - t_i| \leq \tau] $$ where $\tau$ is a temporal tolerance.

The proposed framework shifts video evaluation from coarse accuracy metrics to a detailed diagnostic of where temporal reasoning fails, enabling targeted model improvements for event bookkeeping.