TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning

Overview§

TurnSight addresses a core challenge in Tool-Integrated Reasoning (TIR): sparse, trajectory-level credit assignment in long-horizon tool-use tasks. Instead of relying on ground-truth answers or external skill retrievers, TurnSight generates supervision from execution-conditioned hindsight — i.e., using the actual states visited during rollouts to construct turn-level targets. This makes the training signal denser and more aligned with the agent's real behavior.

Method§

TurnSight operates on turn-level trajectories within each episode. For a turn $t$ (state $s_t$, action $a_t$), it estimates hindsight returns $G_t^{(h)}$ for multiple lookahead horizons $h \in \mathcal{H}$. For example:

$$G_t^{(h)} = \sum_{k=t}^{t+h-1} \gamma^{k-t} r_k + \gamma^h V(s_{t+h})$$

where $V$ is a value estimate based on the actual execution path. These multi-horizon views capture progress at different levels of granularity.

Cross-Horizon Directional Agreement. For a supervision signal to be reliable, the signs of $G_t^{(h)}$ should agree across horizons. TurnSight keeps a target only if:

$$\text{sign}(G_t^{(h)} - V(s_t)) \text{ is consistent for all } h \in \mathcal{H}$$

Otherwise the target is discarded, preventing noisy or contradictory feedback from dominating the update.

Normalization Across Sibling Rollouts. Reliable hindsight targets are then normalized across all rollouts that share the same starting state (sibling rollouts). This yields a scalar signal $\beta_t$ that encodes the relative quality of the current turn:

$$\beta_t = \frac{G_t^{(h^*)} - \mu_{\text{sibling}}}{\sigma_{\text{sibling}} + \epsilon}$$

Adaptive Advantage Modulation. The final policy-gradient loss uses the original RL advantage $A_t$ (e.g., GAE) modulated by $\beta_t$, but preserves the sign of $A_t$:

$$\mathcal{L}_{\text{TurnSight}} = -\mathbb{E} \left[ \log \pi_\theta(a_t|s_t) \cdot A_t' \right], \quad A_t' = \begin{cases} A_t \cdot (1 + \lambda \cdot \beta_t) & \text{if } \beta_t > 0 \\ A_t & \text{otherwise} \end{cases}$$

This avoids biasing the optimization direction while amplifying or dampening the advantage based on hindsight quality.

Implementation Sketch§

def turnsight_loss(samples, horizon_returns, advantages, sibling_stats):
    """
    samples: list of turn tuples (state, action, value)
    horizon_returns: dict mapping turn idx -> {h: hindsight_target}
    advantages: original RL advantages (e.g., GAE)
    sibling_stats: dict of mean/std for turn across sibling rollouts
    """
    losses = []
    for i, (state, action, value) in enumerate(samples):
        # Cross-horizon agreement check
        signs = {h: (horizon_returns[i][h] - value).sign() for h in horizons}
        if all(signs[h] == signs[horizons[0]] for h in horizons):
            # Normalize across siblings
            beta = (horizon_returns[i][h] - sibling_stats[i]['mean']) / \
                   (sibling_stats[i]['std'] + 1e-8)
            if beta > 0:
                advantage = advantages[i] * (1.0 + lambda_ * beta)
            else:
                advantage = advantages[i]
        else:
            advantage = advantages[i]
        losses.append(-log_prob(action, state) * advantage)
    return torch.stack(losses).mean()

Why It Works§

• Dense supervision: every turn gets feedback, not just the final outcome.
• State-relevant targets: hindsight views come from actual trajectories, avoiding mismatch with ground-truth or retrieved skills.
• Noise reduction: cross-horizon agreement filters unreliable targets.
• Stable policy updates: sibling normalization and sign preservation keep RL optimization grounded.

TurnSight is evaluated on three TIR benchmarks and shows consistent improvements over trajectory-level RL baselines, demonstrating the value of turn-level, execution-conditioned hindsight for tool-use agents.