AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design

Overview§

AutoDesign frames multimodal source-to-structured-output tasks (e.g., academic paper-to-poster) as a long-horizon agentic process. It introduces a meta-harness optimizer that iteratively refines the agent's harness (code/config/prompts) based on rollouts, enabling recursive self-improvement.

Core Methodology§

Harness Definition§

A harness $H$ is a program/configuration that wraps a base LLM agent, defining its tools, control flow, and prompts. The goal is to maximize a task-specific score $S(H)$ by executing rollouts on a benchmark.

Meta-Harness Optimization Loop§

AutoDesign maintains a meta-harness optimizer (a strong LLM) that proposes iterative edits to the harness $H_t$ based on past rollout feedback. The update rule resembles a policy improvement step:

$$ H_{t+1} = \text{Opt}\big(H_t,\, \{(\text{rollout}_i, \text{score}_i)\}_{i=1}^{k}\big) $$

where the optimizer generates code changes (diffs) and the harness is re-evaluated. The process is fully autonomous: the optimizer selects actions, executes them via a code agent, and observes results.

System Architecture§

• Code Agent: Executes the harness, performs tool calls (e.g., reading PDFs, calling vision-language models), and produces the poster.
• Meta-Harness Optimizer: Maintains a memory of rollout outcomes and generates harness improvements (e.g., prompt templates, workflow logic, error-handling routines).

A simplified training loop:

# Pseudo-PyTorch-style loop for harness optimization
opt_agent = MetaOptimizer()  # LLM-based
code_agent = CodeAgent()

for epoch in range(max_epochs):
    # Evaluate current harness on a batch
    scores = []
    for paper in sample_benchmark(batch_size=4):
        result = code_agent.run(harness=H, input=paper)
        scores.append(score(result))
    feedback = summarize_scores(scores)

    # Meta-optimizer proposes harness update (diff/patch)
    H_new = opt_agent.propose_update(H, feedback)

    # Apply update and keep if improves
    new_scores = evaluate(H_new, sample_benchmark(batch_size=4))
    if mean(new_scores) > mean(scores):
        H = H_new

Key Components§

PosterBench§

• Main Track: 100 papers across 5 disciplines.
• PosterBench-mini: shared 10-paper subset for controlled evaluation.
• Scoring combines layout, content relevance, and visual clarity (auto-metric + human study).

DesignHarness§

After optimization, AutoDesign produces DesignHarness, a reusable harness that can be plugged into different base models. Integrating it improved average score from 54.99 to 67.39 (+12.4%) across 7 code-agent-model configurations.

Autonomous Loop§

In a fully autonomous run, AutoDesign executed 253 tool calls and 11 editing turns in 40 minutes at < $3 cost, reaching average conference-poster quality in human evaluation.

Results§

• 78.32 on PosterBench Main Track, surpassing Claude Design by 7.45 points.
• System-blind human study shows highest human preference among evaluated systems.
• The meta-harness optimizer's ability to accumulate experience is the decisive factor; static harnesses underperform.

Why It Matters§

AutoDesign demonstrates that agent design itself can be optimized end-to-end using a meta-loop, moving beyond hand-crafted agent workflows. This is a step toward self-improving agent systems that align with human aesthetic/functional priors via iterative empirical feedback.