UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

Overview§

UniClawBench addresses the limitations of existing agent benchmarks by focusing on five core capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. It comprises 400 bilingual real-world tasks executed in live Docker containers, with step-by-step completion checkpoints.

Evaluation Strategy§

The closed-loop evaluation involves three agents:

• Executor Agent: The agent under test, which performs the task.
• Hidden Supervisor Agent: An LLM that grades each step using predefined checkpoints.
• User Agent: Simulates multi-turn human feedback without revealing grading criteria.

This setup enables dynamic, realistic evaluation while preventing score leakage.

Technical Details§

Tasks are designed to isolate specific capabilities. For example, Long-Context Reasoning tasks require the agent to process lengthy documentation (e.g., a 10-page API manual) to extract relevant information. Cross-Platform Coordination tasks involve interacting with multiple services (e.g., Slack and email) to complete a workflow.

The benchmark uses a scoring function based on completion ratio: $$ \text{Score} = \frac{1}{N} \sum_{i=1}^{N} c_i $$ where $c_i \in \{0,1\}$ indicates completion of checkpoint $i$.

Implementation Snippet§

The following pseudocode illustrates the evaluation loop:

def evaluate_agent(agent, task, checkpoints):
    env = DockerEnvironment(task.container_config)
    user_agent = UserAgent(task.user_profile)
    supervisor = SupervisorAgent(checkpoints)

    for step in range(max_steps):
        observation = env.get_observation()
        action = agent.act(observation, user_agent.history)
        env.step(action)
        user_agent.update_history(observation, action)
        
        if supervisor.is_completed(env.state):
            break

    return supervisor.score(env.history)

Capability Disentanglement§

By comparing models (e.g., GPT-4, Claude) across multiple agent frameworks (e.g., ReAct, Plan-and-Solve), UniClawBench reveals how base model capabilities and framework choices jointly impact performance. For instance, a model with strong Long-Context Reasoning may fail under a framework that truncates context.

Key Results§

• Skill-diverse tasks show that no single model excels across all capabilities.
• Frameworks with explicit planning (e.g., ReAct) improve Exploration tasks but hinder Skill Usage for models with weak instruction-following.
• The bilingual design reveals language-specific biases in Multimodal Understanding (e.g., Chinese text in screenshots).

Conclusion§

UniClawBench provides a fine-grained, capability-oriented evaluation for proactive agents in real-world settings, enabling researchers to pinpoint weaknesses in both models and frameworks.