ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Overview§

ExtractBench is a benchmark for schema-guided enterprise document extraction, targeting agentic systems that must faithfully follow user-defined schemas and produce source-grounded outputs. It evaluates 4,869 pages across 370 documents, 8 domains, and 67 types, with challenge tags. The benchmark uniquely combines accuracy, completeness, grounding, and cost into a single evaluation framework.

Dataset and Curation Pipeline§

The ground truth is curated via three complementary strategies:

1. Independent-system agreement for real-world documents (multiple extraction systems must agree on values).
2. Known values for synthetic document lists, enabling exact ground truth.
3. Human verification for forms, where humans validate extracted fields.

This hybrid approach scales while maintaining high-quality labels, balancing the difficulty of real documents with the precision of controlled synthetic data.

Evaluation Metrics§

Three primary metrics are defined:

• Value F1: Order-insensitive token-level F1, computed as

$$\text{Value F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$ where precision = correctly extracted fields / total extracted fields, recall = correctly extracted fields / total ground-truth fields, and order-insensitivity allows records to be matched regardless of sequence.

• Grounding F1: Two variants—word-level and page-level F1—measure how well extracted values are traced to source evidence.
• Record completeness: The fraction of required records successfully extracted, crucial for long multi-record documents.
• Measured cost: Dollar cost per correct extraction, enabling efficiency comparisons.

Key Findings§

• Commercial VLMs perform well on short documents but often truncate record lists in long documents, leading to low completeness.
• Coding agents (e.g., using code-based extraction) retain higher accuracy on long documents but at significantly higher cost.
• LlamaExtract Agentic Plus ranks first across all three core metrics, achieving accuracy comparable to coding agents while being far cheaper.

The full leaderboard and code are publicly available.

Example Evaluation Code§

The Python snippet below sketches how to compute order-insensitive value F1 for a single document:

from collections import Counter

def value_f1(predicted_values, true_values):
    """Calculate order-insensitive token F1."""
    pred_counter = Counter(predicted_values)
    true_counter = Counter(true_values)
    
    # True positives are the overlapping tokens
    tp = sum((pred_counter & true_counter).values())
    precision = tp / len(predicted_values) if predicted_values else 0
    recall = tp / len(true_values) if true_values else 0
    
    if precision + recall == 0:
        return 0.0
    return 2 * precision * recall / (precision + recall)

# Example usage
extracted = ["invoice#123", "$500"]
ground_truth = ["$500", "invoice#123"]  # order differs
print(value_f1(extracted, ground_truth))  # output: 1.0

ExtractBench provides a robust, multi-metric standard for comparing extraction agents, highlighting the trade-off between accuracy, grounding, and cost in real-world document workflows.