Handover of In-Context Learning State Across Session Boundaries

Problem and Formulation§

This paper addresses a practical yet under-theorized problem: when an LLM session ends (due to context limits, restart, or agent handoff), what information must be preserved for the task to continue correctly in a new session? The authors formalize this as handover of a task-relative in-context learning (ICL) state. Let the original session contain task $T$, data $D$ (examples/evidence), and the continuation procedure (e.g., prompts and constraints). The new session sees a handover record $H$ and a query $Q$, and must produce a distribution over outputs $Y$. The goal is to find $H$ such that the predictive distribution is preserved: $P(Y \mid H, Q, T) = P(Y \mid D, Q, T)$.

A key distinction is made between exact recovery of earlier context and preservation of the target distribution. Under an exogeneity condition (the query distribution is independent of the data given the task), predictive equivalence becomes the appropriate criterion. This allows the authors to characterize the coarsest deterministic sufficient handover as a mapping from the original context to an equivalence class that induces the same output distribution for every query.

Core Theory: Predictive Equivalence and Bits§

Define the ICL state $s = \sigma(D, T)$ as the minimal statistic that determines the output distribution for any query. The handover $H$ is sufficient if it is a function of $s$ (or $D,T$) that preserves $\sigma$. The coarse-grained equivalence relation $H \sim H'$ when $\mathbb{E}_Q \mathrm{KL}(P_Y \| P'_Y) = 0$ yields the coarsest deterministic sufficient handover. The number of equivalence classes determines the fixed-length bit requirement: $b = \log_2 |\mathcal{H}_{\mathrm{min}}|$.

An important finding is the cost of writing before the query is known. If the writer must produce $H$ without seeing $Q$, it may need to store more information than if it could tailor $H$ to $Q$. The paper quantifies this as a minimax regret term, separating the effects of the memory constraint, the writer's strategy, and the continuation procedure.

Three-Part Handover Record§

The authors propose a concrete handover structure that balances exactness and compression:

1. Exact record of decisions and constraints (e.g., user instructions, chosen hyperparameters, retrieved documents). These are often small but must be preserved verbatim.
2. Task-justified sufficient statistics for repeated evidence (e.g., mean/covariance for Gaussian data, counts for discrete distributions). These compress redundant information.
3. Original observations whose effect is not captured by the statistics (outliers, data points that materially change the posterior). This ensures the handover is still sufficient when the statistics are approximate.

This three-part design is analogous to a lossless compression scheme with a residual term.

Theoretical Results and Memory-Performance Trade-offs§

For Gaussian linear regression $Y = X\beta + \epsilon$, the handover state is finite-dimensional: the sufficient statistic is the pair $(X^\top X, X^\top y)$. The paper shows this is an exact handover and gives finite-bit perturbation bounds: if the statistics are quantized to $m$ bits, the resulting predictive error is bounded by $O(2^{-m/d})$ where $d$ is the parameter dimension.

For nonparametric regression with squared error loss, the authors derive minimax bounds linking memory to accuracy. If the handover uses $m$ bits, the best achievable expected prediction error is $\Theta(m^{-2/d})$ for smoothness $d$, matching the usual nonparametric rate with memory playing the role of sample size. This reveals a fundamental trade-off: memory is a resource as critical as data.

Implementation Sketch§

Below is a simplified PyTorch-style illustration of the three-part handover for a Gaussian linear task.

import torch
import math

class HandoverState:
    def __init__(self):
        self.constraints = []        # part 1: exact constraints
        self.suff_stat_x = None      # part 2: X^T X
        self.suff_stat_xy = None     # part 2: X^T y
        self.outliers = []           # part 3: residual observations

    def add(self, x, y, is_constraint=False, is_outlier=False):
        x = x.unsqueeze(0)
        if is_constraint:
            self.constraints.append((x, y))
            return
        # update sufficient statistics
        if self.suff_stat_x is None:
            self.suff_stat_x = x.T @ x
            self.suff_stat_xy = x.T * y
        else:
            self.suff_stat_x += x.T @ x
            self.suff_stat_xy += x.T * y
        # outlier detection (simplified: residual > threshold)
        if self.suff_stat_x is not None:
            beta = torch.linalg.solve(self.suff_stat_x, self.suff_stat_xy)
            resid = (y - x @ beta).item()
            if abs(resid) > 1.0:
                self.outliers.append((x, y))

    def to_bits(self, quantile_bits=8):
        # quantize statistics to finite bits
        qx = torch.quantize_per_tensor(self.suff_stat_x, scale=1e-3, zero_point=0, dtype=torch.qint8)
        qxy = torch.quantize_per_tensor(self.suff_stat_xy, scale=1e-3, zero_point=0, dtype=torch.qint8)
        return {"constraints": self.constraints, "qx_int": qx.int_repr(), "qxy_int": qxy.int_repr()}

The actual paper does not provide code, but this sketch captures the essential logic: maintain exact constraints, accumulate statistics, and retain residuals that statistics cannot summarize.

Takeaways§

The paper gives the first rigorous theory for session handover in LLM applications. Its three-part record is directly actionable: practitioners should preserve instructions verbatim, compress repeated evidence into sufficient statistics, and keep outliers. The memory-error bounds provide guidance for allocating storage budgets in long-running or multi-agent systems.