TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

Overview§

TestEvo-Bench addresses the lack of grounded evaluation for test automation agents. It provides real-world test and code co-evolution tasks mined from git commits, with two tracks: test generation (writing new tests for new behavior) and test update (adapting failing tests to code changes). Each task includes a full executable environment, enabling metrics like pass rate, coverage, and mutation score. The benchmark is "live": tasks are timestamped and periodically added, allowing cutoff-based evaluation to reduce data leakage.

Task Construction§

From 59,950 candidate co-evolution records across 152 open-source Java projects, 746 test generation and 509 test update tasks were curated. Each task corresponds to a commit where both production code and test code changed. For test generation, the agent receives the code before and after the change and must write new tests; for test update, the agent gets the commit's diff and must fix failing tests.

Evaluation Metrics§

• Pass rate: Fraction of tasks where all tests pass.
• Coverage: Line/branch coverage of the changed code by generated/updated tests.
• Mutation score: Fraction of killed mutants (injected faults) by the tests.
• Cost: API call cost per task (limited to a budget).

Agents are scored overall as success rate (pass rate on all tasks).

Agents Evaluated§

Four state-of-the-art agent harnesses: Claude Code, Gemini CLI, SWE-Agent, each paired with strong foundation models (Claude Opus 4.7, Gemini 3.1 Pro). Agents have a budget per task (e.g., $5 for Gemini CLI, $10 for Claude Code).

Results§

• Test generation: best agent achieves 77.5% success rate.
• Test update: best agent achieves 74.6% success rate.
• Performance drops by ~20% on the most recent tasks (post-2024), indicating potential data leakage.
• Under limited budget (e.g., $1 per task), success rates plummet to ~30%.

Implementation Details§

Tasks are stored as Docker containers with all dependencies. The agent receives:

• A prompt describing the code change (diff or before/after snapshots).
• Access to the repository file system.
• A test execution command.

The agent iteratively writes tests, runs them, and fixes errors until tests pass or budget exhausted.

Python Example: Loading and Running a Task§

import subprocess
import json

def run_task(task_dir: str, agent_script: str) -> dict:
    """Run a TestEvo-Bench task.
    
    Args:
        task_dir: Path to task directory with Dockerfile, code, and config.
        agent_script: Path to agent script that takes task_dir as argument.
    """
    config = json.load(open(f"{task_dir}/task.json"))
    print(f"Running {config['track']} task: {config['task_id']}")
    result = subprocess.run(
        ["docker", "run", "--rm", "-v", f"{task_dir}:/task", 
         "testevo-agent", "python", agent_script, "/task"],
        capture_output=True, text=True, timeout=600
    )
    return {
        "passed": result.returncode == 0,
        "output": result.stdout,
        "coverage": parse_coverage(result.stdout)
    }

def parse_coverage(output: str) -> float:
    # Extract JaCoCo coverage report
    import re
    match = re.search(r'Line coverage: ([0-9.]+)%', output)
    return float(match.group(1)) if match else 0.0

Key Equations§

Success rate ($S$) for a set of tasks $T$: $$ S = \frac{1}{|T|} \sum_{t \in T} \mathbb{I}[\text{test suite passes on task } t] $$

Mutation score ($M$): $$ M = \frac{\# \text{killed mutants}}{\# \text{total mutants}} $$

Coverage ($C$): $$ C = \frac{\text{lines covered by tests}}{\text{lines changed}} \times 100\% $$

Conclusion§

TestEvo-Bench provides a realistic, execution-grounded benchmark for test automation agents. Current agents show promise but struggle on recent data and under cost constraints, highlighting room for improvement.