SocietyBench: Forecasting Counterfactual Social-World Evolution

Overview§

SocietyBench is a new benchmark that measures an LLM's ability to understand and forecast real-world social events, rather than just completing isolated tasks. It converts a one-line event topic into a date-indexed timeline of factual events and public opinion, then generates audited forecasting questions at every cut-off date. Critically, before evaluation, a three-phase anonymization procedure strips named entities and shifts dates, creating a counterfactual social world that preserves structural realism but prevents the model from pattern-matching pre-training knowledge.

Benchmark Construction§

• Data Collection: Web news and social media posts are gathered from five platforms across Chinese and English editions.
• Timeline Distillation: Factual events and public-opinion layers are separated, resulting in a date-indexed structure.
• Counterfactualization: Each named entity is replaced and every date is shifted by a per-event constant. This yields structurally identical but surface-different timelines.
• Question Bank: Every cutoff date yields a forecasting question with a ground-truth outcome, forming 125 prediction points across 5 events.

Evaluation Metrics§

Two orthogonal axes are scored, each from 0 to 100:

1. Probability Calibration — Measures how well predicted probabilities match empirical frequencies. A Brier score is computed:

$$B = \frac{1}{N}\sum_{i=1}^N (p_i - o_i)^2$$ where $p_i$ is the predicted probability and $o_i$ is the binary outcome. This is converted to a calibration score: $$\text{CalScore} = 100 \times (1 - B)$$

1. Temporal Accuracy — Measures how close the predicted timing is to the actual event time. For a predicted time $\hat{t}$ and actual time $t$, the temporal score is:

$$\text{TempScore} = 100 \times \frac{1}{N}\sum_{i=1}^N \max\left(0, 1 - \frac{|t_i - \hat{t}_i|}{T}\right)$$ where $T$ is a normalization horizon.

The two axes can decouple: a model may be well-calibrated but poor at timing, or vice versa.

Python Scoring Snippet§

The following snippet illustrates the dual-axis scoring:

import numpy as np

def societybench_scores(pred_probs, actuals, pred_times, actual_times, horizon):
    # Probability calibration (Brier-based)
    brier = np.mean((pred_probs - actuals) ** 2)
    cal_score = 100 * (1 - brier)
    
    # Temporal accuracy
    temp_scores = np.maximum(0, 1 - np.abs(pred_times - actual_times) / horizon)
    temp_score = 100 * np.mean(temp_scores)
    
    return cal_score, temp_score

Results & Key Findings§

• The strongest of six frontier LLMs achieves only 75.0/100, versus a trivial anchor of 50.
• Calibration and temporal accuracy are largely independent, meaning single-axis evaluation is misleading.
• Three agent frameworks built on the same base model fail to improve over the base, suggesting current agent scaffolding does not add social-forecasting ability.
• Two model-free heuristics trail every LLM.
• Per-event gaps reach 21.4 points on a single axis, demonstrating the necessity of multi-event evaluation.

All anonymized timelines, question banks, ground truth, and scoring code are publicly released.