The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents

Overview§

This paper introduces a framework for evaluating LLM agents by separating the net effect of adding procedural skills into gains (tasks solved only with skills) and regressions (tasks solved only without skills). Analysis over ~6,000 runs across two benchmarks shows that regressions are substantial and the primary differentiator between skills is how little they regress, not how many new tasks they solve. Three causes of regression are identified: skill description osmosis, grounding displacement, and verification displacement.

Methodology§

Agents are compared in paired runs: with skills ($A_{+s}$) and without skills ($A_{-s}$). Outcomes are categorized:

• Gain: task solved by $A_{+s}$ but not $A_{-s}$
• Regression: task solved by $A_{-s}$ but not $A_{+s}$
• Residual failure: failed by both
• Shared success: solved by both

The net effect of a skill is: $$ \text{Net} = \text{Gain} - \text{Regression} $$

Regression Analysis§

Three regression modes:

1. Skill description osmosis: The presence of skill descriptions in the prompt alters behavior even when not invoked.
2. Grounding displacement: The skill's procedure overrides how the agent interprets inputs.
3. Verification displacement: The skill suppresses output checks the agent would otherwise perform.

Persistent Failures§

Analysis shows most failures stem from grounding and verification issues, not lack of procedural guidance. Skills overemphasize procedure while under-supporting grounding and verification.

Code Snippet (Pseudo-Python for evaluation)§

import json

def evaluate_skill(agent, skill, tasks):
    results_with = []
    results_without = []
    for task in tasks:
        agent_with = agent.copy()
        agent_with.add_skill(skill)
        success_with = agent_with.run(task)
        results_with.append(success_with)
        
        agent_without = agent.copy()
        success_without = agent_without.run(task)
        results_without.append(success_without)
    
    gains = sum(1 for w, wo in zip(results_with, results_without) if w and not wo)
    regressions = sum(1 for w, wo in zip(results_with, results_without) if not w and wo)
    net = gains - regressions
    return {"gains": gains, "regressions": regressions, "net": net}

Key Findings§

• Best skills outperform others by regressing less, not by gaining more.
• Many regressions and persistent failures are recoverable through better grounding and verification.
• Evaluation should report gains and regressions separately, not just aggregate improvement.

Implications§

When designing skills for LLM agents, prioritize minimizing regressions over maximizing successes. Focus on supporting grounding and verification in addition to procedural guidance.