Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Synopsis§

This paper introduces a cost-aware evaluation framework for language-model security agents, moving beyond traditional peak success rate metrics. The authors evaluate both offensive (Cybench CTF challenges) and defensive (Splunk BOTS v1 SOC investigations) agents across multiple models at varying inference budgets and tool usage costs. Key findings: offensive performance scales with additional test-time compute, and open-weight models (e.g., Llama-3-70B) can approach proprietary systems (e.g., GPT-4) cost-effectively; defensive performance is more sensitive to disciplined tool use and telemetry navigation rather than raw reasoning budget. The work argues for economic efficiency as a core metric in security agent benchmarks.

Core Methodology§

The evaluation decomposes success into two cost components: inference cost (tokens generated) and tool cost (API calls, environment interactions). For each model and challenge, they measure success/failure at multiple cost budgets. They introduce a cost-success curve plotting success rate as a function of total budget (in USD). The budget is computed as:

$$C_{\text{total}} = \alpha \cdot \text{inference\_tokens} + \beta \cdot \text{tool\_calls}$$

where $\alpha$ and $\beta$ are cost coefficients (e.g., $\alpha = \$0.01/1K$ tokens for GPT-4, $\beta = \$0.001$ per API call). For defensive tasks, additional costs like telemetry query fees are included. They evaluate models including GPT-4, Claude-3, Llama-3-70B, and Mixtral-8x22B.

Implementation Details§

The agent architecture for both offensive and defensive tasks is a ReAct-style loop (Reason + Act). The agent receives observations, reasons, then outputs a tool call or final answer. Tool calls are executed in a sandboxed environment. The paper uses a budget-aware stopping criterion: the agent stops when its cumulative cost exceeds a threshold. Below is a simplified PyTorch-like pseudo-code snippet illustrating the cost-aware agent loop:

class CostAwareSecurityAgent:
    def __init__(self, model, cost_coeffs):
        self.model = model
        self.inference_cost_per_token = cost_coeffs['inference']
        self.tool_cost_per_call = cost_coeffs['tool']
        self.total_cost = 0.0

    def run(self, task, max_budget):
        observation = task.initial_observation
        while self.total_cost < max_budget:
            # Reason
            tokens = self.model.generate(observation)
            self.total_cost += len(tokens) * self.inference_cost_per_token
            # Act
            action = parse_action(tokens)
            if action['type'] == 'tool':
                result = execute_tool(action)
                self.total_cost += self.tool_cost_per_call
                observation = result
            elif action['type'] == 'answer':
                return action['content'], self.total_cost
        return None, self.total_cost  # exceeded budget

Key Results§

• Offensive tasks: Scaling inference budget improves success for all models. Open-weight models (e.g., Llama-3-70B) at higher budgets match GPT-4 at lower budgets, making them cost-competitive.
• Defensive tasks: Increasing inference budget alone yields diminishing returns. Success correlates with the agent's ability to selectively query telemetry (e.g., focusing on suspicious events) rather than exhaustive enumeration.
• Cost decomposition: Tool calls dominate costs in defensive tasks (~80%), while inference dominates offensive tasks (~70%).

Implications§

The paper advocates for benchmark designs that report performance at multiple budget levels, akin to cost curves in economics. It suggests that defensive agent improvements should focus on better tool orchestration and telemetry triage, not just model scale.