Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning

Problem and Motivation§

Large language models (LLMs) exhibit a failure mode called _repetitive copying_ in long-context reasoning tasks: they extensively copy text from the input into their reasoning traces instead of generating novel reasoning. This behavior intensifies with context length and is linked to insufficient grounding in task-relevant evidence. The paper diagnoses that models copy indiscriminately from both key evidence and irrelevant distractor context, and those that fail to focus on key evidence are more likely to answer incorrectly.

Method: GEAR (Grounding Evidence-Aware Reward)§

GEAR augments the standard accuracy-based reward in reinforcement learning (RL) with two additional signals:

• Grounding reward $R_{\text{ground}}$: rewards overlap between the reasoning trace and key evidence.
• Distractor penalty $R_{\text{distract}}$: penalizes overlap with irrelevant context.

Formally, the total reward is: $$R = R_{\text{acc}} + \alpha R_{\text{ground}} - \beta R_{\text{distract}}$$ where $R_{\text{acc}}$ is the accuracy reward (1 for correct, 0 otherwise), and $\alpha, \beta$ are scaling hyperparameters.

The overlap is measured via a token-level Jaccard similarity between the reasoning trace and the evidence/distractor sets. Let $T$ be the set of tokens in the reasoning trace, $E$ the set of key evidence tokens, and $D$ the set of distractor tokens. Then: $$R_{\text{ground}} = \frac{|T \cap E|}{|T \cup E|}, \quad R_{\text{distract}} = \frac{|T \cap D|}{|T \cup D|}$$

To enable this on natural-language data without manual annotation, the authors develop an automated pipeline that constructs evidence-annotated datasets from arbitrary documents. The pipeline uses: 1. An LLM (e.g., GPT-4) to generate a set of question-answer pairs from the document. 2. For each question, the LLM extracts the minimal supporting evidence from the document (key evidence) and a set of irrelevant sentences from elsewhere in the document (distractors).

Training Details§

GEAR is applied during RL fine-tuning (e.g., using PPO) of a base LLM. The policy is initialized from a supervised fine-tuned (SFT) model. For each training step, a batch of prompts is sampled, the model generates a reasoning trace and answer, and the reward is computed using both accuracy and the grounding terms. Loss is the standard PPO clip objective.

A simplified PyTorch-style pseudocode for the reward calculation:

def compute_reward(trace_tokens, evidence_tokens, distractor_tokens, is_correct, alpha=0.5, beta=0.5):
    trace_set = set(trace_tokens)
    evidence_set = set(evidence_tokens)
    distractor_set = set(distractor_tokens)
    
    r_acc = 1.0 if is_correct else 0.0
    
    # Jaccard similarity for grounding
    intersect_ev = len(trace_set & evidence_set)
    union_ev = len(trace_set | evidence_set)
    r_ground = intersect_ev / union_ev if union_ev > 0 else 0.0
    
    # Jaccard similarity for distractor penalty
    intersect_di = len(trace_set & distractor_set)
    union_di = len(trace_set | distractor_set)
    r_distract = intersect_di / union_di if union_di > 0 else 0.0
    
    total_reward = r_acc + alpha * r_ground - beta * r_distract
    return total_reward

Experiments and Results§

GEAR is evaluated on three long-context reasoning benchmarks (e.g., HotpotQA, 2WikiMultihopQA, MuSiQue) across model scales (LLaMA2-7B, 13B, 70B). Key results:

• Consistent improvements over standard RL (accuracy-only reward) of up to +4.6 average points.
• Gains are larger at longer contexts (e.g., +6.3 points for 16K-token context vs +2.1 for 4K).
• Reduces repetitive copying: the fraction of reasoning trace tokens copied from the prompt decreases (e.g., from 35% to 22%).
• Reduces thinking length (total tokens generated) by 10-15%.

Ablation studies confirm that both the grounding reward and distractor penalty contribute, and the automated pipeline produces annotations close to human quality.

Conclusion§

GEAR demonstrates that explicit reward shaping for grounding can mitigate repetitive copying and improve long-context reasoning. The automated pipeline makes it scalable. The findings suggest that accurate grounding in relevant evidence remains a crucial and under-optimized capability for long-context LLMs.