WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

Overview§

This paper introduces WorldCup Arena, a prospective evaluation benchmark that measures the forecasting ability of six frontier LLMs over the 39 days of the 2026 FIFA World Cup. Unlike retrospective benchmarks that must filter out memorized answers, this evaluation is leakage-free by construction: every question is asked before the event occurs. The frozen archive contains 4,494 scored predictions across 104 matches, 12 group winners, and a pre-tournament outright pool.

Evaluation Design§

Before each kickoff, each LLM was asked to fill a seven-market prediction card for the upcoming match: three-way outcome (home win/draw/away win), over/under goals, both teams to score, correct scoreline, and other markets. All models had extended thinking and native server-side web search. Because no answer existed on the web at the time of prompting, memorisation is impossible and filtering is unnecessary.

Scoring Methodology§

Predictions were scored using standard proper scoring rules. For each match, the three-way outcome probabilities $\mathbf{p} = [p_{home}, p_{draw}, p_{away}]$ were compared against the one-hot encoded actual outcome $\mathbf{o}$. The Brier score is:

$$B = \frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} (p_{ik} - o_{ik})^2$$

with $K=3$. Accuracy is simply $\arg\max_k p_{ik} = o_i$. The paper reports an average outcome accuracy of 63.9%, which is statistically level with a strategy of always backing the bookmaker's favourite — exactly what the models usually do.

Key Findings§

The study documents several shared behaviours:

• Agreement without accuracy: Models agree with one another far more often than they are right. A majority vote among the six provides no improvement over the average individual.
• Compressed distributions: They under-commit to draws and total goals, and their scoreline predictions cluster onto a single prototypical result, e.g. 1-0 or 2-0.
• Accuracy tracks lopsidedness, not information: Performance is best on heavily favoured teams and worst on close ties, where the briefing dossiers are richest. This suggests the models are not effectively exploiting qualitative information.
• No sharp differentiation: The standings are stable at the top and bottom, but the middle churns; overall margins between systems are narrow.

Code Illustration§

The following Python snippet shows how a single match prediction is scored, which is the core evaluation loop used to build the benchmark archive.

import numpy as np

def score_match(pred_probs, actual_outcome):
    """
    pred_probs: array of shape (3,) for [home, draw, away]
    actual_outcome: int in {0, 1, 2}
    Returns (accuracy, brier_score)
    """
    predicted_class = int(np.argmax(pred_probs))
    accuracy = 1 if predicted_class == actual_outcome else 0
    one_hot = np.eye(3)[actual_outcome]
    brier = float(np.sum((pred_probs - one_hot) ** 2))
    return accuracy, brier

# Example: model predicts 45% home, 30% draw, 25% away
example_probs = np.array([0.45, 0.30, 0.25])
actual = 0  # home win
acc, br = score_match(example_probs, actual)
print(f"Accuracy: {acc}, Brier: {br:.4f}")

The authors release the full briefing dossiers, fixtures, official results, and scoring code as a benchmark, allowing future models to be evaluated on the same frozen predictions.