Automated Discovery Has No Universally Superior Harness

Technical Summary§

Core Problem§

Automated discovery systems (e.g., OpenEvolve, TTT-Discover) combine multiple design choices (archives, parent selection, exploration, budget allocation) into a single recipe. Evaluating these harnesses is expensive and stochastic, leading to unreliable comparisons. The paper demonstrates that no fixed harness dominates across model-problem pairs, motivating online adaptation.

Methodology§

• Decomposition: OpenEvolve-style evolutionary search and TTT-Discover are broken into components: archive strategy (none, random, quality-diversity), parent selection (random, tournament, fitness-proportional), exploration (crossover, mutation, LLM-based), and budget allocation (fixed, adaptive).
• Evaluation: 30 budget-matched harnesses × 12 model-problem pairs (e.g., GPT-3.5, LLaMA on math reasoning, code generation) using >3.1M LLM rollouts. Each run repeated with independent trials for statistical rigor.
• Key Finding: No harness consistently bests others; variants of OpenEvolve underperform simpler alternatives (e.g., random search with mutation). Early discovery progress strongly correlates with final performance ($\rho \approx 0.85$).

Adaptive Allocation Algorithm§

Leveraging early progress, the authors propose an adaptive procedure:

1. Start $K$ harnesses with equal budget $B/K$ (partial budget).
2. After $T$ steps, evaluate each harness's performance and prune the bottom fraction.
3. Reallocate remaining budget to surviving harnesses proportionally to their performance.

Formally, let $s_i^{(t)}$ be a score (e.g., best objective so far) for harness $i$ at time $t$. After pruning threshold $\tau$, survivors $S = \{i : s_i^{(t)} > \tau\}$. Remaining budget $B_\text{rem}$ is allocated: $$b_i = \frac{\exp(\beta s_i^{(t)})}{\sum_{j \in S} \exp(\beta s_j^{(t)})} \cdot B_\text{rem}$$

Implementation Code Snippet§

import numpy as np

def adaptive_allocation(harness_scores, budget_remaining, beta=1.0, prune_fraction=0.3):
    """
    harness_scores: dict {harness_id: score}
    budget_remaining: float, total remaining budget
    Returns: dict {harness_id: allocated_budget}
    """
    n = len(harness_scores)
    k_prune = int(n * prune_fraction)
    sorted_harnesses = sorted(harness_scores.items(), key=lambda x: x[1], reverse=True)
    survivors = sorted_harnesses[:n - k_prune]  # keep top (1-prune_fraction)

    scores = np.array([s for _, s in survivors])
    weights = np.exp(beta * scores)
    weights /= weights.sum()

    allocations = {}
    for (hid, _), w in zip(survivors, weights):
        allocations[hid] = w * budget_remaining
    return allocations

Results§

Adaptive allocation outperforms both committing to a random fixed harness and a non-adaptive ensemble (equal budget split), achieving higher median performance and lower variance. The paper releases run pools with null distributions for future comparisons.

Implications§

Harness selection should be treated as a hyperparameter tailored to model and problem. Online adaptation using early signals is a promising direction for automated discovery.