APEX-Accounting

APEX-Accounting: Benchmarking LLMs as Autonomous Accountants§

Overview§

APEX-Accounting is a closed benchmark designed to test whether frontier LLMs can perform real-world accounting work. It consists of 160 tasks across 10 distinct "worlds", each simulating a company's accounting system including spreadsheets, PDFs, and transactional data. Tasks include reconciling accounts, accruing expenses, posting journal entries, and generating financial reports. The benchmark targets agentic capabilities: models must interact with files, execute multi-step reasoning, and adhere to accounting standards.

Benchmark Design§

• Worlds: Each world contains a complete accounting system (e.g., QuickBooks-like environment) along with source documents. Models are given task instructions and access to files.
• Tasks: 160 tasks authored and solved by professional accountants. Each task has a graded rubric with multiple criteria (e.g., correct account, correct amount, proper documentation).
• Evaluation Protocol: Models are given a token budget (ranging from $1 to $50) to complete tasks. They can make multiple API calls or use tools. The harness records all model interactions and scores responses against the rubric.

Metrics§

Two primary metrics are reported:

• Mean Criteria@3: For each task, the model is allowed up to 3 attempts (or retrieves top-3 responses). The fraction of grading criteria satisfied by the best attempt is computed, then averaged over all tasks. This measures partial success.
• Pass^8 / Pass@8: These strict metrics require all criteria to be met. Pass^8 (possibly meaning passing all 8 criteria) and Pass@8 (pass rate given 8 attempts) are reported. The abstract shows very low scores (e.g., 2.6% for Pass^8, 21.5% for Pass@8), indicating the difficulty of full autonomous accuracy.

Key Findings§

• Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, followed by Muse-Spark-1.1 (xHigh) at 52.6%.
• Scaling token budget from $1 to $50 increases overall scores, but a Simpson's paradox emerges: within a fixed budget, tasks where models spend more tokens have lower pass rates. This suggests that harder tasks consume more budget but remain unsolved, inflating the aggregate score when budget is increased.
• No model achieves high strict pass rates, implying that current LLMs are not yet reliable for fully autonomous accounting work.

Simpson's Paradox Analysis§

Let $S(b, t)$ be the score on task $t$ with budget $b$. The aggregate score $\bar{S}(b) = \frac{1}{T}\sum_t S(b, t)$. Increasing $b$ from $1$ to $50$ raises $\bar{S}$ because each task's score tends to increase. However, if we condition on token usage $u_t(b)$ within a fixed budget $b$, tasks with higher $u_t(b)$ have lower $S$. This is because for a given $b$, tasks requiring many tokens are inherently harder and less likely to succeed. The paradox: $\frac{\partial \bar{S}}{\partial b} > 0$ but $\frac{\partial S}{\partial u} < 0$ within each $b$.

Code Snippet: Pass@k Calculation§

Below is a Python snippet to compute the Pass@k metric (for top-k attempts). Assuming each task produces $n$ candidate responses and $c$ are correct (satisfy all criteria), Pass@k is:

$$\text{Pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

import numpy as np

def pass_at_k(n, c, k):
    """
    Compute Pass@k for a single task.
    n: total number of generated responses
    c: number of correct responses (all criteria met)
    k: number of top attempts considered
    """
    if n - c < k:
        return 1.0
    return 1.0 - np.prod([(n - c - i) / (n - i) for i in range(k)])

# Example: if n=10, c=2, k=8
print(pass_at_k(10, 2, 8))  # -> 0.9778

For the Mean Criteria@3 metric, a more detailed rubric scoring is needed, but Pass@k captures the strict all-or-nothing evaluation.

Conclusion§

APEX-Accounting provides a realistic and challenging testbed for agentic AI in finance. The low strict pass rates highlight the gap between current LLMs and expert accountants. The Simpson's paradox offers insights into budget allocation: simply spending more tokens does not guarantee solving hard tasks.