The Bitter Lesson of Tool Calling

Overview§

This paper empirically compares programmatic tool calling (PTC) against traditional JSON tool calling in large language models. Instead of having the model emit structured JSON to invoke a tool, PTC exposes tools as typed Python stubs and lets the model write executable code. The code can chain and parallelize calls naturally, reducing overhead and enabling more flexible agent behavior. The evaluation uses BFCL v4 (Berkeley Function Calling Leaderboard v4) across 14 current and prior model generations.

Core Methodology§

In JSON tool calling, the model is given a schema and must output a strict JSON object like {"function": "search", "arguments": {"query": "LLM"}}. PTC instead provides a Python function signature:

def search(query: str, limit: int = 10) -> list[dict]:
    """Search the web. Returns list of results."""

The model generates Python code that imports and calls these functions, e.g. results = search("LLM", limit=5). The executor runs the code in a sandbox, collects results, and returns them to the model in the same agent turn. This allows natural control flow (for loops, conditionals) and parallel fan-out.

The paper defines the primary comparison as the task success rate $S$ on BFCL v4. For each model $M$, they measure $S_{\text{PTC}}$ and $S_{\text{JSON}}$, then compute the delta $\Delta = S_{\text{PTC}} - S_{\text{JSON}}$.

Implementation Details§

The evaluation uses 14 models spanning multiple generations (GPT-5.6 family, GPT-4o, Claude, Llama, etc.). They also test two robustness conditions:

• Parallel fan-out: the model must invoke multiple independent tools simultaneously. PTC naturally emits concurrent.futures-style code or simple list comprehensions.
• Context rot: the model receives a long irrelevant preamble before the task, testing degradation. JSON baselines degrade an average of 2.3%, while PTC holds stable.

The execution engine is a sandboxed Python interpreter that handles imports, timeouts, and result serialization. A minimal implementation would look like:

import ast, exec

TOOLS = {"search": search_func}

def execute_code(code: str) -> str:
    tree = ast.parse(code)
    allowed = {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)}
    env = {"__builtins__": {"__import__": restricted_import}, **TOOLS}
    exec(code, env)
    return env.get("last_result") or ""

Results§

PTC matches or exceeds JSON tool calling in 11 of 14 models on the base BFCL v4 benchmark. The GPT-5.6 family shows the largest improvement at +10.6%, suggesting that stronger code-capable models benefit most from writing tool calls as code. Under parallel fan-out, PTC matches or outperforms baseline in 13 of 14 models. Under context rot, JSON baselines drop by 2.3% on average, while PTC remains stable, implying that code generation is less susceptible to instruction forgetting.

Significance§

The paper's title alludes to the "bitter lesson"—replacing rigid, hand-engineered JSON schemas with general-purpose code execution scales with model capability. PTC is a viable alternative for building agentic systems, particularly for models with strong coding ability. The results also highlight that benchmark design should account for execution conditions such as parallelism and context length.

Conclusion§

Programmatic tool calling is a robust, efficient, and increasingly superior alternative to JSON tool calling for LLM agents. As models improve, the advantage of code-as-tools will likely grow, making it a key design choice for next-generation agent frameworks.