QuoteBench: How Matched Scores Can Hide Command-Path Failures

Overview§

QuoteBench quantifies the boundary between an LLM coding agent's command-generation ability and the execution transport (serialization, wrapping, reparsing) that sits between the model and the shell. It uses 56 one-shot tasks drawn from 14 incident-derived families, with exact final-state validation. The central claim is that "matched" execution scores conflate generation errors with transport-induced failures, and that a single aggregated score is not an intrinsic model property.

Methodology§

Each task has a raw reply $r$ generated by the model under a generation contract $G$ (e.g., plain bash). The same reply is then replayed through an execution path $E$ that may include an added parser—here, a deliberately unescaped single-quote interpolator—which simulates a common real-world failure mode. The task defines a final-state validator $V$ that checks whether the state after execution matches the goal.

For any configuration, the success probability can be decomposed as: $$\Pr(\text{success}) = \Pr(\text{raw reply correct}) - \Pr(\text{raw correct but transport breaks}) + \Pr(\text{raw incorrect but transport fixes}).$$

The paper measures the gap between matched execution score (through the full transport) and the raw generation score, and then the recovery when the transport boundary is disclosed to the model. The key trick is that escaping at the interpolation point (e.g., inserting a backslash before the unescaped parser) reproduces the raw-path outcome for each replayed reply, making the transport deterministic and reversible.

Key Results§

Replaying identical replies through the added parser reduces success by 55.4–73.2 percentage points across eight same-window configurations. Disclosing the boundary recovers 30.4–60.7 points for six configurations, but zero or slightly negative for two, indicating that some models cannot adapt even when the failure mode is known. Raw generation is nearly saturated at the frontier, so transport adaptation is the main differentiator. For example, GPT-5.6-sol shows a matched gap of only −3.6 points, but that hides −64.3 points of damage and +60.7 points of compensation.

Implications for Evaluation§

Model rankings change when the deployment configuration changes: there is one unambiguous reversal among 26 comparable pairs and four more that hinge on a single task. Thus, the paper argues that any evaluation of command-issuing agents must report: model configuration, generation contract, execution path, operating point, and final-state validator. A matched score alone is meaningless.

Minimal Evaluation Snippet§

# Illustrative protocol: generate, replay, validate
def evaluate(model, task, transport):
    raw_reply = model.generate(task.prompt)          # generation contract
    executed = transport(raw_reply)                   # e.g., unescaped parser
    raw_ok = task.validator(task.state, raw_reply)
    matched_ok = task.validator(task.state, executed)
    disclosed_reply = model.generate(task.prompt + transport.disclosure)
    disclosed_ok = task.validator(task.state, transport(disclosed_reply))
    return raw_ok, matched_ok, disclosed_ok

Conclusion§

QuoteBench provides a controlled, replay-based methodology to separate model-level command generation from transport-level failures. It exposes systematic gaps that matched scores hide, and demonstrates that deployment-specific adaptation is a key capability that should be measured and reported explicitly.