Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search

Overview§

This paper identifies and measures a critical gap in how retrieval systems are evaluated for multi-step agentic tasks. Traditional static relevance metrics (e.g., BM25, reader-based scores) assume a document's utility comes solely from directly answering the current query. In contrast, agentic search involves sequential reasoning and tool use, where a document may be causally valuable not for its immediate answer but for enabling the next search action (e.g., surfacing a discriminative entity). The authors formalize this discrepancy via Counterfactual Trajectory Utility (CTU) and show it is nearly uncorrelated with Static RAG Utility (SRU).

Methodology§

Static RAG Utility (SRU)§

\[ \text{SRU}(d, q) = f(\text{reader}(q, d) - \text{reader}(q, \emptyset)) \] where $f$ is a scoring function (e.g., exact match improvement).

Counterfactual Trajectory Utility (CTU)§

For each document $d$ read during an agent's trajectory $\tau$, a counterfactual trajectory $\tau_{-d}$ is run by deleting $d$ and replaying the rest from that point. CTU aggregates three deltas:

• $\Delta_{\text{answer}}$: change in final answer quality (e.g., F1)
• $\Delta_{\text{retrieval}}$: change in the next query's retrieval quality (e.g., recall of relevant items)
• $\Delta_{\text{turns}}$: change in number of turns (penalizing extra turns)

\[ \text{CTU}(d) = \alpha \Delta_{\text{answer}} + \beta \Delta_{\text{retrieval}} - \gamma \Delta_{\text{turns}} \]

Bridge Documents§

Documents with low SRU but high CTU (top-right quadrant when axes are normalized). The paper reports ~30% of documents fall into this "bridge cell".

Key Results§

• Spearman $\rho = -0.026$ between CTU and SRU over 23,322 observations.
• Using BM25 + cross-encoder as reader proxy still yields 27.2% bridge cell.
• Mechanism: Observable Entity Relevance (OER) shows that discriminative entities (present only in relevant documents) appear in the agent's next query 4.02× more often than non-discriminative entities.

Implications§

Optimizing for static relevance does not transfer to agentic retrieval. Instead, systems should be trained to generate documents that provide actionable next-step entities.

Code Illustration§

import copy
from typing import List, Callable

def compute_ctu(agent, question: str, trajectory: List[Dict], doc_idx: int, 
                answer_quality_fn: Callable, retrieval_quality_fn: Callable) -> float:
    """
    Compute CTU for a single document at doc_idx in the trajectory.
    """
    # Original outcome
    original_answer = trajectory[-1]['output']
    original_retrieval = trajectory[doc_idx]['retrieval_score']
    original_turns = len(trajectory)
    
    # Counterfactual: delete document and replay
    truncated = trajectory[:doc_idx]
    agent_cf = copy.deepcopy(agent)
    agent_cf.memory = [m for m in agent.memory if m.doc_id != trajectory[doc_idx]['doc_id']]
    try:
        cf_trajectory = agent_cf.run(question, initial_memory=truncated)
    except:
        # If agent fails, assign worst outcome
        cf_answer = ""
        cf_retrieval = 0
        cf_turns = 100
    else:
        cf_answer = cf_trajectory[-1]['output']
        cf_retrieval = cf_trajectory[doc_idx]['retrieval_score']
        cf_turns = len(cf_trajectory)
    
    # Deltas
    delta_answer = answer_quality_fn(original_answer) - answer_quality_fn(cf_answer)
    delta_retrieval = retrieval_quality_fn(original_retrieval) - retrieval_quality_fn(cf_retrieval)
    delta_turns = original_turns - cf_turns  # fewer turns is better
    
    # Weighted combination (turns delta is inverted sign)
    ctu = delta_answer + delta_retrieval - delta_turns
    return ctu

Equations§

Static utility: $\text{SRU}(d,q) = \mathbb{E}_{\text{reader}}[\text{score}(q, d) - \text{score}(q)]$

Causal utility: $\text{CTU}(d) = \Delta_{\text{answer}} + \Delta_{\text{retrieval}} - \Delta_{\text{turns}}$

Bridge condition: $\text{SRU}(d) < \tau_{\text{low}}$ and $\text{CTU}(d) > \tau_{\text{high}}$.