Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Technical Overview§

Hindcast addresses the evaluation of LLM forecasting ability by preventing two forms of data leakage: temporal leakage (models trained on data that includes the outcome) and retrieval leakage (retrieval of post-event reports). The method replays resolved Polymarket prediction markets using a frozen snapshot of Reddit data, with a per-market cutoff date $t_0$. The model only accesses posts before $t_0$, and its forecasts are compared both to the actual outcome and to the market's own price at $t_0$ (a human forecast from the same information).

Core Methodology§

For each market with resolution date $t_{\text{res}}$, a cutoff $t_0 < t_{\text{res}}$ is chosen (e.g., when the market opened). A static Reddit dump is filtered to include only posts with timestamp $\leq t_0$. The LLM is given a prompt with the market question and the filtered posts, and outputs a probability $p_{\text{LLM}}$. The score is the Brier score: $$\text{BS} = \frac{1}{N}\sum_{i=1}^N (p_i - o_i)^2$$ where $o_i \in \{0,1\}$ is the outcome. Additionally, comparison to market price $p_{\text{market}}$ highlights whether the model adds value.

Implementation Details§

Key design choices:

• Static snapshot: Reddit comments from a fixed dump (e.g., 2023) prevent contamination from later updates.
• Per-market cutoff: Each market uses its own $t_0$, ensuring the model never sees post-event information.
• Retrieval: Models can use retrieval-augmented generation (RAG) on the snapshot; performance gains are only legitimate if Reddit contained relevant pre-event discussion.

Simplified Evaluation Loop§

import datetime
from typing import List, Dict

def evaluate_market(model, market: Dict, reddit_snapshot: List[Dict]):
    cutoff = market["opened_at"]  # e.g., datetime(2022, 1, 1)
    # Filter snapshot to posts before cutoff
    relevant_posts = [p for p in reddit_snapshot if p["timestamp"] < cutoff]
    prompt = f"Market: {market['question']}\nReddit posts:\n" + "\n".join(p["text"] for p in relevant_posts)
    prob = model.predict_probability(prompt)
    actual = market["outcome"]  # 0 or 1
    brier = (prob - actual) ** 2
    market_price = market["price_at_cutoff"]
    return brier, abs(prob - market_price)

Key Findings§

• Once leakage is closed, retrieval still helps but only when Reddit contained relevant pre-event discussion. Where the archive held only speculation, retrieval actually harms performance (overfitting to noise).
• The method is future-proof: new markets can be added without stale evaluations (same snapshot).
• Human forecasters (market prices) outperform most LLMs, highlighting room for improvement.

Conclusion§

Hindcast provides a robust, contamination-free benchmark for LLM forecasting, revealing that current models rely heavily on leaked information. The frozen snapshot and per-market cutoff are critical to measuring genuine predictive ability.