Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design§

Core Problem§

Professional graphic design is a long-horizon agentic task: structured, editable artifacts emerge from many interdependent actions, and critically, there is no reliable programmatic oracle to score final outcomes. This is the classic "noisy, unverifiable feedback" regime. The paper's central claim is that a frozen frontier model can continually adapt without weight updates or human labels, by accumulating an external procedural memory of natural-language skills.

Methodology: Widening + Deepening§

A frozen LLM operates professional design software through more than 230 tools. A memory bank $\mathcal{M} = \{s_1, \dots, s_n\}$ of natural-language skills starts from 76 documentation-derived skills. Each round applies two complementary mechanisms:

1. Widening (acquisition). For recurring uncovered subtasks, the memory acquires new procedures. Formally, given the set of failed trajectories, cluster uncovered subtasks and synthesize skills: $$\mathcal{M}_{t+1} = \mathcal{M}_t \cup \{\, \text{synth}(u) : u \in \text{Uncovered}(\mathcal{T}_t^{\text{fail}}) \,\}$$

2. Deepening (revision). Existing procedures are revised against their own successful and failed executions: $$s_i' = \text{revise}(s_i, \{(x_j, y_j, \text{outcome}_j)\}_{j \in \text{exec}(s_i)})$$

Matched Replay Gate (the key control)§

Because outcomes are unverifiable, naive edits can regress previously working behavior. The matched replay gate admits a change only if it repairs failures without regressing observed successes: $$\text{accept}(s_i \to s_i') \iff \text{fix}(\mathcal{F}) > 0 \;\wedge\; \text{break}(\mathcal{S}) = 0$$ where $\mathcal{F}$ and $\mathcal{S}$ are matched failing and succeeding replays. This is the critical stabilization primitive that makes noisy continual adaptation safe.

Experimental Results§

• 5 rounds over 1,406 real user briefs and 1,869 auto-graded trajectories, no weight updates, no human labels.
• Memory grows 76 → 139 skills.
• GenEval2 execution success on Claude-Sonnet-4: 72.7% → 99.3% (+11.99 generation-quality points).
• 61.8% / 67.6% win rates vs. no-skill agent across four specialized benchmarks (Sonnet-4 / Opus-4.6).
• Ablation: on 200 held-out user-traffic briefs, widening or deepening alone reaches 49.4% / 48.6% win rate; combined reaches 58.5% (p = 0.025) — the two mechanisms are synergistic.

Implementation Sketch§

A simplified loop showing memory growth and the replay gate:

memory = load_doc_skills()  # 76 seeds
for round in range(5):
    trajectories = rollout_agent(policy=frozen_llm,
                                 tools=DESIGN_TOOLS_230,
                                 briefs=user_briefs,
                                 skills=memory)
    # --- Widening ---
    uncovered = mine_recurring_uncovered(trajectories.failures)
    for u in uncovered:
        memory.add(synthesize_skill(policy, u))

    # --- Deepening with matched replay gate ---
    for s in list(memory):
        candidate = revise_skill(policy, s,
                                 successes=trajectories.success(s),
                                 failures=trajectories.failure(s))
        fixes   = replay(candidate, trajectories.failure(s))
        breaks  = replay(candidate, trajectories.success(s))
        if fixes.ok and not breaks.regressed:   # accept only net-positive
            memory.replace(s, candidate)

    print(f"round {round}: {len(memory)} skills")

Why It Matters§

This offers a practical route to continual adaptation under noisy, unverifiable feedback — no fine-tuning, no labels — by treating skills as versioned code under test via matched replay. The synthesis/revision gate is a general pattern applicable to any tool-using agent where a programmatic oracle is unavailable.