Open-Source Benchmarks for Document Extraction Vendor Evaluation

Document extraction has become one of the most consequential inputs to AI agents. When an agent acts on a badly extracted field, it doesn't fail quietly. It can book the wrong invoice total, flag the wrong claimant, or miss a row in a loan schedule that a human reviewer never sees, because the system reported high confidence when it was wrong. That chain of consequence means a company's choice of document extraction vendor sits upstream of every downstream agent workflow it builds, not inside a procurement spreadsheet.

The benchmark ecosystem has grown to match that weight. What used to be a question of raw OCR accuracy now spans schema-guided extraction, long-record completeness, visual grounding, confidence scoring, and cost at production volume. A vendor can post a strong number on a narrow benchmark and still fail in production. You need to know what a given benchmark actually measures before it gets used to justify an infrastructure choice. Three categories of evaluation framework exist for that purpose: schema-guided extraction benchmarks, document layout and structure benchmarks, and end-to-end pipeline benchmarks built around specific high-risk use cases. Each section below works through one piece of that landscape.

From fixed templates to schema-guided extraction

For years, document information extraction benchmarks only worked against a fixed set of pre-built templates. That approach measures something real, but it can't show how a system handles a schema nobody wrote until last week, and that is exactly where most enterprises live. Businesses write a new extraction schema for nearly every new workflow they stand up. A system tuned to one fixed template has no way to prove it will hold up on the next one. Vendors who still lead with template-matched accuracy numbers are answering a question enterprises stopped asking years ago, and that gap should disqualify them on its own.

Schema-guided extraction asks a harder, more honest question. A document and a user-defined schema go in as input, and the agent has to follow that schema faithfully, producing the correct output along with source evidence as grounding metadata, as ExtractBench defines the task. Schema-guided benchmarks expose whether a system can read an instruction and apply it correctly to a document it has no prior familiarity with, not whether it remembers a template. The practical failures this reveals are specific: missing rows buried in long lists, picking the wrong instance of a fact that appears sparsely across a document, overfilling dense forms with noise, confusing similar-looking dates or identifiers or dollar amounts, and the ordinary perception problems of scan noise, handwriting, hierarchical headers, text that continues across a page break, and tables that don't fit a clean grid.

Coverage and limits of the leading open-source benchmarks

No open-source benchmark today accounts for the full range of what a production extraction system needs to get right. Each one does one or two things well and leaves the rest unmeasured, a structural feature of how benchmarks get built. A vendor evaluation resting on a single benchmark number is incomplete by design. Treating one benchmark as sufficient is the most common mistake in procurement conversations, and it's the one worth correcting first.

ExtractBench, published on arXiv, covers 370 documents across 4,869 pages, spanning 8 business domains and 67 distinct document types. It evaluates long-record completeness, real scans and handwriting rather than clean synthetic text, word- and page-level grounding, and measured cost, and it is the only benchmark in its own comparison table that covers all four of those dimensions at once. Those four things rarely get tested together: a benchmark built around clean documents won't show anything about handwriting, and a benchmark that scores accuracy without cost leaves half the infrastructure decision unanswered.

The World Bank's layout detection benchmark asks a different question. It formalizes "data snapshot extraction," the task of identifying and localizing semantically meaningful visual artifacts, figures and tables carrying reusable analytical content, inside humanitarian reports, World Bank policy research working papers, and project appraisal documents. It scores object detection quality and spatial extraction quality jointly, and it releases the source PDFs, the annotation dataset, metadata, and the code behind it. Its central finding is blunt: open-source layout detection models that perform well on conventional academic benchmarks struggle to generalize to real institutional documents, with common failures including confusion between analytical and non-analytical content and the fragmentation of a single composite chart or table into disconnected pieces.

A third benchmark, built around a high-risk public sector use case, evaluates the end-to-end performance of open-source OCR engines, LLMs, and VLMs on student applications for an international study program, a zero-shot, multi-step extraction pipeline that the EU AI Act classifies as high-risk. Vision-language models generally beat OCR-plus-LLM pipelines, but even the strongest open-source models struggled badly: the vast majority of configurations tested scored below an F1 of 0.25. Table extraction splits along its own line within this landscape. Some benchmarks score table quality holistically, judging the overall structure produced. Others use a unit-test approach: they apply binary pass-fail rules at the level of individual cells. These two paradigms can rank systems differently, and that design choice deserves scrutiny before anyone trusts a ranking built on it.

The specific extraction failure modes that benchmark scores can obscure

A single aggregate accuracy number hides the exact failure modes that decide whether a pipeline can be trusted in production. These failure modes appear only at scale, or only under specific document conditions, so a headline score averages them away instead of surfacing them.

Long-record completeness is probably the most consequential of these. A system can read every value on a page correctly and still truncate a long schedule partway through, and the rows simply drop from the output. Most benchmarks don't penalize this the way a production system needs them to, because catching a missing row requires measuring what isn't there, not just scoring what is present. ExtractBench addresses this directly: it includes synthetic long lists built on real document layouts, exposing truncation behavior that shorter test documents never trigger.

Visual grounding gets underweighted the same way. When a long schedule gets truncated and rows go missing, a human reviewer needs a way to trace each extracted field back to where it came from on the page, so the error can be found and fixed quickly. The public sector benchmark adds a finding that applies across all of this: how well an OCR stage preserves document structure is a critical factor on its own, regardless of how capable the downstream LLM or VLM is. If a strong model sits behind an OCR stage that fragments tables and loses spatial relationships, the model can't compensate for the lost structure. OCR quality has to be judged separately from model quality, not folded into one combined score.

Why cost and latency belong inside the benchmark

When companies process documents at these volumes, a cost difference of one cent per page can decide whether an AI initiative is worth running. A benchmark that scores accuracy but not cost is an incomplete tool for vendor evaluation, no matter how carefully it measures everything else. ExtractBench treats cost as a first-class benchmark criterion, the only schema-guided benchmark among its comparisons to do so, on the premise that accuracy and cost have to be judged together: the tradeoff between them is the actual decision an infrastructure team is making.

Latency works the same way, but the right target depends entirely on the workflow. A single reported latency number scores systems with completely different operational profiles the same way, and that can mislead a buyer in either direction. Hybrid processing strategies route documents differently depending on complexity or confidence, so they can land on different cost-accuracy tradeoffs for the same document corpus. A benchmark needs to measure cost alongside accuracy rather than after it, because that tradeoff is the infrastructure decision itself.

Benchmark saturation and domain mismatch

Even the best-built benchmarks run under controlled conditions on a fixed set of documents. The gap between that controlled set and a company's actual document corpus is the main reason a benchmark score doesn't translate directly into a production prediction. Anyone leaning on a published number to make a purchasing decision needs to account for that gap first.

Standard benchmarks tend to lean on cleaner, more uniform documents than production workflows ever produce. The institutional document benchmark states this outcome directly: models that do well on conventional academic document benchmarks fail to generalize to operational institutional documents, and the failure traces to a mismatch between the benchmark's documents and the deployment's documents, not to some inherent weakness in the models themselves.

Benchmark saturation compounds the problem over time. A benchmark that differentiated vendors meaningfully when it was published becomes less informative as those same vendors optimize their systems specifically to score well on it. A result from an older benchmark run may just reflect tuning against that one dataset, not any generalizable gain in production capability. None of this argues for throwing benchmarks out. It argues for building an evaluation protocol that accounts for these limits instead of treating a single published number as the final word.

Building an evaluation protocol that holds up under production conditions

A defensible vendor evaluation combines published benchmark coverage with testing on documents drawn from your own production distribution, because no published benchmark will ever match your documents exactly. Published benchmarks give you a baseline and a vocabulary for comparison. Testing on real documents confirms that baseline holds for the mix of layouts, languages, and document conditions a workflow actually produces.

Benchmarks should be chosen by the capability gap they close, not by overall ranking. ExtractBench is the right tool for evaluating schema-guided accuracy, long-record completeness, grounding, and cost together in one pass. If a workflow touches institutional or policy documents carrying embedded analytical tables and figures, the World Bank layout benchmark is the right one to use. The public sector pipeline benchmark's methodology is the right template for evaluating a full OCR-plus-LLM or VLM pipeline under zero-shot conditions for a high-risk application. Confidence scoring deserves its own place in the protocol: every extracted field should carry a confidence score, and the evaluation needs to check whether those scores are calibrated. A system that reports high confidence on a wrong extraction is more dangerous in production than one that correctly flags its own uncertainty and routes the field to a human. Grounding and traceability need to be scored as a protocol requirement, because tracing every extracted field back to the exact passage it came from matters for human review, and under emerging compliance frameworks it may count as an audit requirement in its own right.

Evaluation needs to happen at the level of the full pipeline, not the model alone. The public sector benchmark found that OCR structural preservation matters independent of downstream model capability. A benchmark that only tests the model misses failures that happen earlier in the chain, so the evaluation has to cover parsing, extraction, and output validation as one connected system. Cost and latency belong in the rubric from the first test run: measure cost per page at the actual volumes a workflow will process, and judge latency against whether the workflow is batch-tolerant or real-time. Document distributions shift and vendors update their models, so the evaluation can't be treated as a one-time gate passed during vendor selection. It needs to run continuously against real production documents, because a benchmark result from the original selection process may say nothing about how the system performs six months later.

Criteria for evaluating a document extraction vendor

Everything above points toward a short, concrete list of what separates a vendor that holds up under real document volume from one that only performs well in a curated demo. How a vendor backs its own accuracy claims comes first. If a vendor publishes open-source benchmarks against production-representative document types, rather than proprietary benchmarks run on hand-picked clean documents, you can check that claim yourself. Disqualify any vendor who won't submit to that kind of outside scrutiny. ExtractBench is a clear example of this kind of benchmark: by jointly covering long-record completeness, real scans and handwriting, word- and page-level visual grounding, and measured cost at production volume, it formalizes the capability matrix an infrastructure decision depends on, past a single accuracy score and into the tradeoffs between accuracy, cost, and latency that decide which vendor survives contact with real-world document variety.

Schema-guided extraction, confidence scoring, grounding, and human-in-the-loop routing should work as parts of one coherent platform rather than components stitched together from separate vendors, because the failure modes this article has walked through tend to appear at the seams between pipeline stages. A schema-guided benchmark like ExtractBench asks a system to generalize across every layout a single invoice schema might encounter, and tuning to pre-built templates cannot get a vendor there. Before you commit to a vendor, confirm that its vision model handles real-world variation, scan noise, and hierarchical document structure together, because that combination is the test that actually predicts production performance.

For regulated workflows, security and compliance posture belongs in the evaluation itself, not in a separate checklist handled later by legal. In healthcare, financial services, and public sector document processing, SOC 2, HIPAA, and GDPR certification, options for self-hosted or bring-your-own-cloud deployment, and zero data retention policies are gating criteria. They rule vendors in or out before anyone talks about accuracy numbers. Developer experience indicators, an API-first design, SDKs across multiple languages, an evaluation suite built into the platform itself, schema optimization tooling, and a short path from integration to production, decide whether a strong benchmark score becomes a deployed system or stays a result sitting in a research paper.