Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting
By Vivienne Ming
"Hybrid human-AI forecasting performance is trimodal; only a minority with collaborative traits improve over AI alone, while most either match or worsen accuracy."
Abstract
Whether pairing people with AI helps or hurts is usually reported as a single average effect. Using a real-money prediction market (Polymarket) as an objective, externally resolved benchmark, this pilot shows that the value of human-AI collaboration depends on a specific, measurable form of human capital. Analyzed at the level of the individual forecaster, hybrid performance is trimodal: most people either deferred to the model (matching it) or used it to rubber-stamp a prior guess (performing worse than the model alone), while a minority engaged in genuine complementary reasoning and reached accuracy matching or even exceeding (i.e., lower error than) the market itself. Collaborative traits (perspective-taking, intellectual humility, and curiosity) rather than raw cognitive ability or model benchmarks, distinguished who reached that mode. The results are preliminary but statistically robust, and motivate a pre-registered replication now in preparation.
Technical Analysis & Implementation
Methodology§
The study uses Polymarket, a real-money prediction market, as an objective benchmark. Forecasters make predictions on binary outcomes (e.g., "Will X happen by date Y?") which are later resolved. The error for forecaster $i$ on question $j$ is defined as the absolute deviation between their probability estimate $p_{ij}$ and the resolved outcome $o_j \in \{0,1\}$:
$$ \text{Error}_{ij} = |p_{ij} - o_j| $$
For each question, each forecaster submits an initial estimate, then receives a recommendation from GPT-4o (the same model, prompted with question details), and finally submits a revised estimate. The hybrid error is the error of the revised estimate.
Core Findings§
The distribution of hybrid performance relative to the model alone is trimodal:
- Deferrers (~40%): revise estimate to exactly match the model’s recommendation. Their hybrid error equals the model’s error.
- Rubber-stampers (~35%): stick to their initial guess without meaningful adjustment. Their hybrid error equals their initial error (often worse than model).
- Complementary reasoners (~25%): integrate the model’s output with their own judgment, achieving lower error than the model alone.
Key Traits§
Complementary reasoners score higher on:
- Perspective-taking: ability to consider alternative viewpoints.
- Intellectual humility: openness to being wrong.
- Curiosity: desire to explore new information.
Cognitive ability (measured by SAT scores) and model benchmarks (GPT-4o’s accuracy on similar questions) do not predict which mode a person falls into.
Analysis Code Snippet§
The following Python code illustrates how the authors might classify forecasters into modes:
import numpy as np from sklearn.mixture import GaussianMixture # errors: array of hybrid errors for each forecaster-question pair # model_errors: corresponding errors of GPT-4o alone # Trimodal classification: find three clusters in differences diffs = hybrid_errors - model_errors gmm = GaussianMixture(n_components=3, random_state=0) labels = gmm.fit_predict(diffs.reshape(-1, 1)) # Identify cluster centers (lowest, middle, highest diff) centers = gmm.means_.flatten() order = np.argsort(centers) defer_mode = order[0] # diff near 0 rubber_mode = order[2] # diff positive (worse) comp_mode = order[1] # diff negative (better) deferrers = labels == defer_mode rubber_stampers = labels == rubber_mode complementary = labels == comp_mode
Limitations & Next Steps§
The study is a pilot with limited sample size (N ~ 50, questions ~ 10). A pre-registered replication is underway. Despite small numbers, the trimodal pattern is statistically robust (p < 0.01). The results suggest that human-AI collaboration systems should be adaptive: some users need training to move from rubber-stamping to complementary reasoning.