Learning When to Trust via Selective Context Preference Optimization

Overview§

The paper addresses the failure mode where LLMs that ignore all context appear robust but are useless when context is trustworthy. The authors formalize the problem as selective trust: a model should trust context when it helps and ignore it when misleading. They introduce MIST, a human-annotated benchmark rendering each reasoning item under four matched conditions: clean, misleading, correct-context, and irrelevant-context. They also propose SC2W (Susceptibility to Context-to-Wrong), a paired metric counting how often a misleading signal flips a clean-correct answer to wrong. They then present SCOPE (Selective Context Preference Optimization), a DPO-based training method that balances preference pairs across all four conditions, reducing SC2W while preserving accuracy on trustworthy contexts.

Core Methodology§

SC2W Metric§

SC2W is computed on paired examples $(x_{\text{clean}}, x_{\text{misleading}})$ that share the same underlying reasoning question but differ in appended context. Given a model $\pi_\theta$, let $y$ be the correct answer. We define:

$$ \text{SC2W}(\pi_\theta) = \mathbb{E}_{(x_c, x_m, y) \sim \mathcal{D}} \left[ \mathbb{1}\big[\pi_\theta(y|x_c) \text{ is correct} \land \pi_\theta(y|x_m) \text{ is wrong}\big] \right] $$

This measures the probability that a misleading context flips a correct answer. The companion metric SC2W-rate normalizes by the number of clean-correct cases.

MIST Benchmark§

Each reasoning item is rendered in four ways:

1. Clean: no extra context.
2. Misleading: a plausible but false context.
3. Correct-context: a relevant true hint.
4. Irrelevant-context: unrelated neutral text.

This matched design enables fine-grained analysis of trust behavior.

SCOPE Training Objective§

Standard DPO optimizes:

$$ \mathcal{L}_{DPO}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right] $$

SCOPE mines failure pairs: examples where the model is clean-correct but misleading-wrong. It creates preference pairs $(y_w = \text{clean-correct answer}, y_l = \text{incorrect answer under misleading})$. Critically, SCOPE balances these pairs so that all four context types appear equally in the batch, preventing overfitting to the misleading case alone. The final objective is:

$$ \mathcal{L}_{SCOPE} = \mathcal{L}_{DPO}^{clean} + \mathcal{L}_{DPO}^{misleading} + \mathcal{L}_{DPO}^{correct} + \mathcal{L}_{DPO}^{irrelevant} $$

where each term is computed on preference pairs drawn from that condition. This preserves performance on helpful and neutral contexts while teaching the model to resist adversarial ones.

Implementation Sketch§

Below is a simplified PyTorch-style pseudocode for constructing SCOPE preference pairs and applying DPO.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def build_scope_pairs(clean_correct, misleading_wrong):
    # clean_correct: list of (question, correct_answer)
    # misleading_wrong: list of (misleading_context, wrong_answer)
    pairs = []
    for (q_c, y_c), (q_m, y_w) in zip(clean_correct, misleading_wrong):
        assert q_c == q_m  # same underlying question
        pairs.append({
            "x_clean": q_c,
            "y_w": y_c,
            "x_misleading": q_m,
            "y_l": y_w,
        })
    # Balance by duplicating/undersampling across conditions
    return pairs

def dpo_loss(model, ref_model, x, y_w, y_l, beta=0.1):
    logits_w = model(x, y_w).logits
    logits_l = model(x, y_l).logits
    with torch.no_grad():
        ref_logits_w = ref_model(x, y_w).logits
        ref_logits_l = ref_model(x, y_l).logits
    log_ratio_w = logits_w.sum(-1) - ref_logits_w.sum(-1)
    log_ratio_l = logits_l.sum(-1) - ref_logits_l.sum(-1)
    loss = -torch.log(torch.sigmoid(beta * (log_ratio_w - log_ratio_l))).mean()
    return loss

# Training loop: for each balanced batch, compute loss over all four conditions

Results§

SCOPE substantially reduces SC2W on popular open-sourced models (e.g., Llama, Mistral) compared to baselines that train only on misleading data. The balanced training avoids the degenerate solution of ignoring all context. Evaluations show accuracy is preserved on clean, correct-context, and irrelevant-context conditions, demonstrating genuine selective trust rather than blanket resistance.

Conclusion§

The paper argues that robustness evaluation should be reframed from "ignoring all context" to "knowing when to trust." The MIST benchmark and SC2W metric provide a standardized way to measure selective trust, and SCOPE offers a practical DPO-based training recipe to improve it.