Can AI agents conduct open-ended AI research? Early evidence from two case studies

Summary§

This paper introduces a novel evaluation method, shadow evaluations, to measure the ability of AI agents to conduct open-ended AI research. Unlike existing benchmarks that focus on narrow, verifiable tasks or rely on noisy peer review, shadow evaluations have an agent tackle the central research question of a high-quality unpublished paper, and the original authors assess the output. The authors ran two such evaluations on unpublished NeurIPS 2026 submissions, providing frontier agents with six days and thousands of dollars in compute. The agents completed all engineering tasks without human assistance but failed to make substantial progress on the research questions, leading to unambiguous rejection by the original authors.

Methodology§

Shadow evaluations involve the following steps:

1. Select a target paper: Choose an unpublished, high-quality paper with a clearly defined open-ended research question.
2. Define the agent task: The agent must attempt to answer the paper's research question, given only the problem description (not the solution). The agent can use any resources, including code, literature, and compute.
3. Run the agent: Deploy a state-of-the-art LLM agent with a scaffold (e.g., tool-use, planning) for a fixed time and compute budget.
4. Evaluate: The original authors review the agent's output using a structured rubric, including criteria such as correctness, novelty, and depth. The agent's work is scored on a 1-5 scale, with a score below 3 indicating rejection.

Key Findings§

In both case studies, the agents achieved perfect scores on engineering components (e.g., implementing baselines, writing code) but scored 1-2 on research insight, novelty, and overall contribution. The authors identified five recurring failure modes:

• Poor judgment about the bar for publishable research: Agents produced trivial or incremental contributions without recognizing their low significance.
• Uncreative responses to shortcomings: When initial experiments failed, agents made minor tweaks (e.g., hyperparameter adjustments) rather than proposing novel approaches.
• Ineffective backtracking: Agents persisted on dead-end paths without recognizing the need to pivot.
• Poor resource awareness: Agents spent disproportionate compute on low-impact experiments.
• Instruction drift: Agents gradually deviated from the original research question, exploring tangential ideas.

A robustness check with a different model and scaffold reproduced these failures, suggesting the issues are fundamental to current agent capabilities.

Mathematical Formulation§

Let $P$ be the target paper with research question $Q$. The agent $A$ operates over a time horizon $T$ with compute budget $C$. The agent's output $O$ consists of a report $R$, code $K$, and experimental results $E$. The original authors evaluate $O$ based on criteria $c_1, c_2, \ldots, c_n$ (e.g., correctness, novelty, clarity) on a scale $[1,5]$. The overall score $S$ is a weighted sum:

$$S = \sum_{i=1}^n w_i \cdot \text{score}(c_i)$$

where $\sum w_i = 1$. A threshold $\theta$ (e.g., $\theta = 3$) determines acceptance if $S \geq \theta$.

Implementation Details§

The agents were built using a frontier LLM (e.g., GPT-4, Claude) with a scaffold that included:

• A file system for code and data.
• A web browser for literature search.
• A Python execution environment.
• A persistent memory for tracking progress.

Agents were given the paper's abstract and a description of the research question, but not the full paper or solution. They were instructed to produce a final report detailing their findings. The budget was $5000 and 6 days per run.

Code Illustration§

Below is a simplified Python script simulating the shadow evaluation loop:

import openai
import subprocess
import time

class ShadowEvaluation:
    def __init__(self, agent, paper_question, budget_dollars, time_days):
        self.agent = agent
        self.paper_question = paper_question
        self.budget = budget_dollars
        self.time_limit = time_days * 86400  # seconds
        self.start_time = time.time()
        self.cost = 0

    def run(self):
        # Agent loop
        while time.time() - self.start_time < self.time_limit and self.cost < self.budget:
            self.cost += self.agent.step(self.paper_question)
        return self.agent.final_report()

# Example usage
eval = ShadowEvaluation(agent=MyAgent(), paper_question="How to improve GAN training stability?", budget_dollars=5000, time_days=6)
report = eval.run()
print(report)

Conclusions§

The paper provides early evidence that current AI agents can handle the engineering aspects of AI research but lack the creativity, judgment, and strategic thinking required for open-ended scientific discovery. The shadow evaluation method offers a scalable and reliable alternative to existing benchmarks.