PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

Overview§

PAST-Bench is a benchmark designed to isolate whether personal AI agents actually improve from retained experience across sessions. It introduces a controlled protocol where each agent runs ordered sequences of fresh-session tasks under matched conditions, toggling retention on/off. The benchmark spans 26 scenarios and 204 episodes, covering four capability dimensions: memory, procedural reuse, information gathering, and update. The authors evaluate seven base models with four agent frameworks and find that improvement is real but uneven, and that identical headline gains can hide differences in whether the intended save → retrieve → update pathway is followed. They then propose Hermes+, an extension of the Hermes agent, adding five targeted interventions across the agent loop, which improves both average gains and pathway evidence.

Core Methodology§

Benchmark Protocol§

The core idea is to compare an agent's performance on a later task with and without prior experience from an earlier task. For each scenario $S$ with task sequence $(T_1, T_2, ..., T_n)$, the agent runs in two modes:

• Retention ON: The agent's memory/tool state persists across sessions.
• Retention OFF: The agent is reset to a baseline state before each task.

The gain for task $T_i$ is defined as: $$ \text{Gain}_i = \text{Score}_{\text{ON}}(T_i) - \text{Score}_{\text{OFF}}(T_i) $$ where $\text{Score}$ is task-specific (e.g., exact match, F1, or human evaluation). To isolate the intended pathway, PAST-Bench also measures structural evidence: does the agent actually save relevant information, retrieve it at the right time, and use it to update its behavior? These are binary indicators $E_{\text{save}}, E_{\text{retrieve}}, E_{\text{update}}$, aggregated into a pathway support score.

Agent Frameworks and Models§

Experiments use four agent frameworks (e.g., LangChain, AutoGPT-like) and seven base LLMs (including GPT-4, Claude, Llama-3, Mistral). Each agent is given the same toolset: a memory store (vector DB), a file system, and a note-taking tool.

Hermes+ Interventions§

Hermes+ builds on the Hermes agent with five interventions targeting distinct stages of the agent loop:

1. Structured Memory Logging: After each task, force the agent to write a structured summary (goal, steps, outputs, pitfalls) to a dedicated memory file.
2. Retrieval-Augmented Re-Prompting: At task start, the system prompt is augmented with relevant past summaries retrieved by semantic similarity.
3. Explicit Update Triggers: The agent is instructed to compare its current behavior with past notes and explicitly update its "task playbook" when it detects contradictory information.
4. Temporal Marking: Every memory entry is timestamped and the agent is required to resolve conflicts by preferring the newest entry.
5. Pathway Reward Shaping: During development, a lightweight reward signal (or instruction in the prompt) encourages the agent to verify it saved/retrieved/updated before executing the task.

The most impactful intervention was the update trigger, especially for scenarios where outdated state must be replaced.

Code Snippet (Illustrative)§

The following pseudocode outlines the benchmark run loop and the Hermes+ memory update mechanism:

import json
from typing import Dict

class HermesPlusAgent:
    def __init__(self, memory_path: str):
        self.memory_path = memory_path
        self.timestamps = {}

    def save_experience(self, task_id: str, summary: Dict, ts: int):
        """Intervention 1: Structured memory logging."""
        record = {"task_id": task_id, "summary": summary, "ts": ts}
        with open(self.memory_path, "a") as f:
            f.write(json.dumps(record) + "\n")
        self.timestamps[task_id] = ts

    def retrieve_relevant(self, query: str, top_k: int = 3):
        """Intervention 2: Semantic retrieval."""
        # In practice, use a vector DB; here we return latest entries.
        with open(self.memory_path, "r") as f:
            lines = f.readlines()[-top_k:]
        return [json.loads(l) for l in lines]

    def update_playbook(self, old_note: Dict, new_info: Dict):
        """Intervention 3/4: Conflict resolution via timestamps."""
        if new_info["ts"] > old_note["ts"]:
            return new_info  # replace outdated state
        return old_note

def benchmark_episode(agent, task_sequence, with_retention):
    scores = []
    for idx, task in enumerate(task_sequence):
        if idx > 0 and with_retention:
            # inject retrieved memories into prompt
            memories = agent.retrieve_relevant(task.query)
            response = agent.run(task, memories)
        else:
            response = agent.run(task, [])
        scores.append(score_task(task, response))
    return scores

Results and Observations§

• Average gains from retention were positive but small (~+5–10% absolute) for most models, with strong variance across capabilities.
• Memory tasks showed the largest gains; update tasks showed the smallest, often negative, as agents failed to overwrite outdated state.
• The pathway analysis revealed that many agents demonstrated gains without actually saving/retrieving—suggesting superficial learning or hidden shortcuts.
• Hermes+ improved average gain by ~2–3x relative to vanilla Hermes on update-heavy scenarios, but the effect was model-dependent (e.g., stronger with GPT-4, minimal with Llama-3).

Implications§

PAST-Bench provides a reproducible methodology for measuring if agents truly improve from experience, not just whether they perform better. Hermes+ offers a practical recipe for engineering persistent self-improvement, though it remains brittle across models. The benchmark invites further research into mechanistic pathway tracing and update-specific training.