It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

Overview§

This paper investigates how linguistic variation in users' expressions of belief (EoBs) influences whether large language models (LLMs) adopt contextual information or rely on their prior knowledge. The authors construct a controlled benchmark by pairing diverse EoBs (e.g., presuppositions, certainty markers, rhetorical questions) with factual statements, enabling precise measurement of context-following behavior across models.

Typology of Expressions of Belief§

EoBs are categorized along four dimensions:

• Form: declarative, interrogative, imperative, etc.
• Evidentiality: direct (e.g., "I saw"), reportative (e.g., "They say"), inferential (e.g., "It seems").
• Epistemic stance: certainty (e.g., "definitely"), uncertainty (e.g., "maybe"), neutrality.
• Tone: formal, informal, sarcastic, etc.

These yield 17 fine-grained types. For each type, a template is constructed, e.g., for presupposition: "As we know, [fact]." Then paired with a world knowledge fact that may be true or false to create a query.

Evaluation Setup§

16 LLMs are tested, spanning architectures (Llama3, Qwen3, Gemma3), scales (1B-30B), and training stages (base vs. instruct). For each EoB-query pair, the model's response is classified as either context-following (adopting the EoB's claim) or knowledge-reliant (contradicting it). The metric is the context-following rate:

$$\text{CFR} = \frac{\# \text{context-following responses}}{\# \text{total pairs}}$$

Key Findings§

• Scale effect: Larger models (e.g., 30B) show lower CFR than smaller ones (e.g., 1B).
• Instruction tuning: Instruct models are less context-following than their base counterparts.
• Persuasive EoBs: Certain types (e.g., rhetorical questions, certainty markers) statistically significantly increase CFR across models.

Implementation Details§

Pairs are generated programmatically using templates and a knowledge base of facts. For each EoB type, a fact is inserted into a template, and the model's response is evaluated via a rule-based classifier or an auxiliary LLM judge.

Example Code Snippet

import json

# EoB templates
templates = {
    "presupposition": "As we know, {fact}.",
    "rhetorical_question": "Isn't it obvious that {fact}?",
    "certainty_marker": "I am absolutely sure that {fact}.",
}

# Generate pairs
eob_pairs = []
for eob_type, template in templates.items():
    for fact in facts:
        eob_pairs.append({
            "type": eob_type,
            "prompt": template.format(fact=fact),
            "target_fact": fact
        })

# Evaluation hook (pseudocode)
def evaluate_model(model, pairs):
    results = []
    for pair in pairs:
        response = model.generate(pair["prompt"])
        is_context_follow = classify(response, pair["target_fact"])
        results.append(is_context_follow)
    return results

The classifier uses string matching or a small model to detect agreement/contradiction.

Conclusion§

This work provides a systematic framework for understanding how linguistic framing affects LLM behavior, highlighting vulnerabilities and offering insights for robust prompt engineering.