Enterprise document automation has a dirty secret: most agents cannot reliably follow a user-defined schema. You hand them an invoice, a purchase order, or a benefits form with a JSON schema describing what to extract, and they return plausible-looking data that fails validation, omits required fields, or hallucinates values with no source grounding.
ExtractBench is a new benchmark that measures what existing agent benchmarks ignore: schema fidelity, record completeness, and source traceability across 4,869 pages of real enterprise documents. The results show that commercial vision-language models truncate long lists, coding agents burn tokens at unsustainable cost, and the gap between demo performance and production reliability remains wide.
Why Schema-Guided Extraction Matters
Schema-guided extraction is the plumbing behind every document automation workflow that feeds downstream systems. The agent receives:
- A document (PDF, scanned image, HTML)
- A user-defined schema (JSON Schema, Pydantic model, OpenAPI spec)
- An instruction to extract structured data that matches the schema
The output must include:
- Values that validate against the schema
- Complete records (no silent truncation)
- Source evidence (page numbers, bounding boxes, or text spans) for each extracted value
This is not summarization. It is not question answering. It is the unglamorous task of turning unstructured documents into structured records that can be inserted into databases, validated by business rules, and audited for compliance.
Existing benchmarks test code generation (HumanEval), question answering (MMLU), or final accuracy on curated tasks. None measure whether an agent can follow a schema with 100% fidelity across hundreds of pages.
The ExtractBench Architecture
ExtractBench evaluates agents across three dimensions:
Value Accuracy: Order-insensitive F1 score comparing extracted values to ground truth. If the schema defines a list of line items, the agent must extract all items regardless of order.
Grounding Metrics: Two levels of source traceability:
- Word-level F1: Does the agent cite the exact text span?
- Page-level F1: Does the agent cite the correct page?
Measured Cost: Token usage and API costs per document, critical for production deployment at scale.
The dataset spans 8 business domains (invoices, contracts, benefits forms, purchase orders) and 67 document types, with clear tags for challenge scenarios:
- Multi-page tables that span page breaks
- Nested schemas with optional fields
- Documents with ambiguous formatting
- Scanned images with OCR noise
Ground truth comes from three sources:
- Independent-system agreement for real documents (two extraction systems must agree)
- Known values for synthetic lists (programmatically generated test data)
- Human verification for edge cases
This hybrid approach balances scale with accuracy. Pure human annotation does not scale to 4,869 pages. Pure synthetic data does not capture real-world formatting chaos.
Evaluation Pipeline
The benchmark runs as a containerized evaluation harness:
# Simplified extraction evaluation loop
def evaluate_agent(agent, document, schema, ground_truth):
# Agent extracts structured data
result = agent.extract(document, schema)
# Validate schema compliance
schema_errors = validate_schema(result.data, schema)
# Compute value F1 (order-insensitive)
value_f1 = compute_value_f1(
predicted=result.data,
ground_truth=ground_truth.data,
ignore_order=True
)
# Compute grounding F1
word_f1 = compute_grounding_f1(
predicted_spans=result.word_spans,
ground_truth_spans=ground_truth.word_spans,
level="word"
)
page_f1 = compute_grounding_f1(
predicted_pages=result.page_refs,
ground_truth_pages=ground_truth.page_refs,
level="page"
)
# Track cost
cost = result.token_count * agent.cost_per_token
return {
"schema_errors": schema_errors,
"value_f1": value_f1,
"word_f1": word_f1,
"page_f1": page_f1,
"cost_usd": cost
}The harness runs each agent against all 370 documents, aggregates scores by domain and document type, and reports percentile distributions for cost.
Results: Where Agents Break
The paper reports results for several agent architectures:
| Agent Type | Value F1 | Word Grounding F1 | Page Grounding F1 | Cost per Doc |
|---|---|---|---|---|
| Commercial VLM (short docs) | 0.87 | 0.72 | 0.89 | $0.12 |
| Commercial VLM (long docs) | 0.61 | 0.54 | 0.71 | $0.18 |
| Coding Agent | 0.91 | 0.83 | 0.94 | $2.40 |
| LlamaExtract Agentic Plus | 0.92 | 0.85 | 0.95 | $0.34 |
Short documents (1-3 pages): Commercial VLMs perform well. They extract values with high accuracy and cite sources reliably.
Long documents (10+ pages): Commercial VLMs truncate record lists. A 15-page invoice with 200 line items returns only the first 50. Value F1 drops to 0.61 because the agent silently omits records without signaling incompleteness.
Coding agents: High accuracy but unsustainable cost. They generate Python scripts to parse documents, run OCR, and validate outputs. Token usage spikes because they iterate on code until tests pass. At $2.40 per document, processing 10,000 invoices per month costs $24,000.
LlamaExtract Agentic Plus: Matches coding agent accuracy at 14% of the cost. The architecture combines a vision model for layout understanding with a smaller language model for schema validation and grounding.
Failure Modes in Production
ExtractBench exposes three failure modes that break production workflows:
Silent truncation: The agent returns a valid JSON object that passes schema validation but omits required records. Downstream systems accept the partial data and make decisions on incomplete information.
Grounding drift: The agent extracts correct values but cites the wrong source page. Auditors cannot verify the extraction, and compliance workflows fail.
Schema hallucination: The agent invents fields not present in the schema or coerces values into the wrong type. A date field receives a string, a required field is null, or a nested object is flattened.
These failures cascade. A truncated invoice list causes payment delays. Grounding drift triggers manual review queues. Schema hallucinations break database inserts and trigger exception handlers.
Deployment Considerations
If you are building document automation on top of agents, ExtractBench suggests several guardrails:
Validate completeness: Do not trust that the agent extracted all records. Compare expected record counts (from document metadata or heuristics) to actual counts. Flag discrepancies for human review.
Enforce grounding: Require the agent to return source evidence for every extracted value. Store grounding metadata in a separate table and surface it in audit UIs.
Test on long documents: Benchmark your agent on documents with 100+ records. If value F1 drops below 0.90, you will face silent truncation in production.
Monitor cost per page: Track token usage and API costs per document type. If cost exceeds $0.50 per document, you cannot scale to enterprise volumes without budget overruns.
Use schema validation as a circuit breaker: Run extracted data through a strict JSON Schema validator before inserting into downstream systems. Reject invalid outputs and route them to a manual queue.
When to Use ExtractBench
ExtractBench is useful if you are:
- Evaluating extraction agents for production deployment
- Building a document automation pipeline that feeds structured data into databases or ERP systems
- Comparing the cost-accuracy trade-off between VLMs, coding agents, and specialized extraction models
- Debugging why your agent works on demo documents but fails on real enterprise PDFs
It is not useful if you are:
- Extracting unstructured summaries or answers (use question-answering benchmarks instead)
- Processing documents with no schema (use retrieval-augmented generation)
- Working with documents under 3 pages (commercial VLMs already perform well)
Technical Verdict
ExtractBench fills a gap in agent evaluation: it measures schema fidelity, completeness, and grounding at enterprise scale. The benchmark reveals that commercial VLMs truncate long documents, coding agents burn unsustainable tokens, and specialized extraction models like LlamaExtract Agentic Plus offer the best cost-accuracy trade-off.
Use this benchmark if you are deploying document automation in production and need to validate that your agent can follow schemas reliably. Avoid it if you are building demos or prototypes where silent truncation and grounding drift are acceptable risks.
The dataset and evaluation harness are open source. Run it against your agent before you commit to a vendor or architecture. The cost of fixing extraction failures in production is orders of magnitude higher than the cost of benchmarking upfront.
Source Links
- ExtractBench Paper (ArXiv)
- Dataset on HuggingFace (link in paper)
- Evaluation Code on GitHub (link in paper)