Validity of LLMs as data annotators: AMALIA on authority
By Manuel Pita
"Proposes recovery gap to test if LLMs use theory-aligned reasoning or shortcuts; finds AMALIA-9B fails for authority construct on Portuguese corpus."
Abstract
A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the question is whether the model follows the construct's theory or reaches the right code by correlated shortcuts. We test this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. If calibration closes that gap, some portability should survive across models and languages; where it does not, the construct-model instrument is the likely locus of failure. We ask whether a calibrated English instrument transfers to AMALIA-9B and to European Portuguese. For one construct and one corpus, it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlates, especially moral outrage near authority figures. An open multilingual LLM closes the gap on the same Portuguese corpus under the same instructions, pointing away from the corpus as the main explanation. AMALIA can still screen and pre-code at scale, but it cannot yet measure this construct well enough to stand alone. The study is a single counterexample, not a verdict on national models; it argues that sovereign-LLM benchmark batteries should test not only agreement with human coders, but the evidential route by which that agreement is warranted.
Technical Analysis & Implementation
Summary§
The paper evaluates whether LLMs can validly annotate theoretical constructs requiring inference, using the moral foundation of "authority" as a test case. It introduces the recovery gap metric to distinguish agreement (reliability) from validity: the difference in performance between a holistic prompt and a decomposed prompt that follows the construct's theory step-by-step. Applied to Portugal's 9B-parameter AMALIA model, the recovery gap is large, indicating reliance on spurious correlations (e.g., moral outrage near authority figures) rather than theory-aligned reasoning. A larger multilingual LLM closes the gap, suggesting the corpus is not the primary cause.
Methodology: Recovery Gap§
Let $\mathcal{P}_{holistic}$ be the prompt that asks the model to directly code the moral foundation of authority from a text. Let $\mathcal{P}_{decomposed}$ be a prompt that breaks the coding into atomic clauses (e.g., presence of authority figure, presence of moral loading, explicit rule to combine) per the construct's codebook. Let $\text{Perf}(\cdot)$ be a performance metric (e.g., exact match F1). The recovery gap is:
$$\text{Gap} = \text{Perf}(\mathcal{P}_{holistic}) - \text{Perf}(\mathcal{P}_{decomposed})$$
A large gap suggests the holistic performance is not attributable to the theory but to surface shortcuts. Calibration (e.g., prompt tuning) aims to close this gap; if it persists, the model-construct instrument is invalid.
Experimental Setup§
- Model: AMALIA (9B parameters, European Portuguese).
- Corpus: Portuguese texts annotated for authority moral foundation by trained humans.
- Comparison: Open multilingual LLM (e.g., Llama 3 8B) on same Portuguese corpus.
- Task: Binary classification: does the text convey authority moral foundation?
Prompts were constructed in English and translated to Portuguese. Decomposed prompts explicitly asked: 1. Is there an authority figure? 2. Does the text express moral approval/disapproval related to authority? 3. Apply the rule: authority = (authority figure present) AND (moral loading related to authority).
Error Analysis§
AMALIA shows high false positives on texts containing "outrage" near authority entities (politicians, police). The recovery gap is ~50% of holistic performance, i.e., decomposed prompt yields only half the F1 score of holistic prompt. The multilingual model shows much smaller gap (~10%), indicating better alignment with the theory.
Code Illustration§
# Simulated recovery gap computation
def recovery_gap(model, texts, holistic_prompt, decomposed_prompt, labels):
holistic_preds = model.predict(texts, holistic_prompt)
decomposed_preds = model.predict(texts, decomposed_prompt)
holistic_f1 = compute_f1(holistic_preds, labels)
decomposed_f1 = compute_f1(decomposed_preds, labels)
gap = holistic_f1 - decomposed_f1
return gap, holistic_f1, decomposed_f1Implications§
- National LLMs cannot be assumed valid for inferential annotation tasks without construct-specific validation.
- Benchmark batteries should include recovery gap tests, not just agreement statistics.
- AMALIA remains useful for screening and pre-coding but not standalone measurement.