Surprisal Theory is Tautological (without Rational Grounding)
By Ryan Cotterell
"Argues surprisal theory is tautological unless language model is rationally grounded; any difficulty pattern can be fit by some model, so empirical tests need non-empirical constraints."
Abstract
Surprisal theory holds that the human processing difficulty of a linguistic unit in context is an affine function of its surprisal under some language model. I argue this claim is a tautology without further constraint: for any non-negative difficulty measure over units in context, there exists a language model whose surprisal is an affine function of it under mild technical conditions. Therefore, because any pattern of difficulty is consistent with some language model, without an additional constraint on the language model, surprisal theory makes no falsifiable predictions. The tautology was long obscured by an assumption implicit in two decades of psycholinguistic work---that the relevant language model is the distribution that generated the training corpus, so that improving corpus fit improves predictions of human behavior. Recent empirical work has undermined this assumption, demonstrating that better corpus models can be worse predictors of processing difficulty. I conclude that breaking the tautology requires a rationalist intervention, i.e., the relevant language model must be derived from a non-empirically motivated model of the comprehender, which could be based on, for instance, memory constraints or processing goals, and that, thus, does not depend on the behavioral data surprisal theory is meant to explain.
Technical Analysis & Implementation
Summary§
Surprisal theory claims that human processing difficulty for a linguistic unit is an affine function of its surprisal under some language model. This paper shows that this claim is tautological: for any non-negative difficulty measure, there exists a language model making the affine relation hold. Therefore, without additional constraints on the language model (e.g., rational principles like memory limits), surprisal theory makes no falsifiable predictions.
Formalization§
Let $D(u, c) \geq 0$ denote processing difficulty of unit $u$ in context $c$. Surprisal theory asserts there exists a language model $P(\cdot|c)$ and constants $a>0$, $b$ such that: $$D(u,c) = a \cdot (-\log P(u|c)) + b.$$
Given any $D$, one can define $P(u|c) = \exp(-D(u,c)/a) / Z_c$ where $Z_c = \sum_v \exp(-D(v,c)/a)$ (assuming finite vocabulary). Setting $b = a \log Z_c$ gives the affine relation. Hence the tautology: any pattern of difficulty is explainable.
Implications§
Empirical work showing better language models (on corpora) sometimes predict human behavior worse undermines the implicit assumption that the relevant language model is the corpus distribution. This gap motivates a "rationalist intervention": the language model must be derived from cognitive constraints (e.g., bounded memory, processing goals) rather than purely fit to data.
Code Example (Conceptual)§
The following Python snippet illustrates constructing a language model from any difficulty function:
import numpy as np
def construct_model(difficulty_func, context, vocab, a=1.0):
# difficulty_func(u, context) returns non-negative number
scores = np.array([difficulty_func(u, context) for u in vocab])
logits = -scores / a
z = np.sum(np.exp(logits))
probs = np.exp(logits) / z
# Then for any unit u: D(u,c) = a*(-log probs[u]) + a*log(z)
# So b = a*log(z)
return probs, a * np.log(z)Conclusion§
The paper reframes the debate: surprisal theory is not falsifiable unless the language model is constrained a priori. Future work should derive language models from rational principles rather than empirically fitting behavioral data.