Regression Testing Frameworks for Document Extraction Accuracy

Document extraction accuracy looks solid in a demo and then falls apart in production, and the reason is almost never the model. It's the absence of a testing framework built to catch drift before a customer does. A snapshot comparison against a fixed reference tells you nothing once that reference goes stale, once a document format shifts, or once a new model version changes its output in ways too subtle for a diff tool to flag. What follows is a working blueprint for building that framework: what to measure, how to build the ground-truth corpus, how to structure the test suite, and how to run the whole thing without it collapsing under its own maintenance load six months in.

Start with the gap that causes most of the trouble. Vendor demos run on sample PDFs: clean, digital-native, generated for the pitch. Production runs on a coffee-stained invoice photographed at an angle, a form with handwritten notes crammed into the margin, a scanned contract rotated 90 degrees, a receipt in a language the parser wasn't tuned for. Legacy OCR systems topped out around 78 to 82% accuracy on semi-structured documents like these. Current multimodal extraction systems clear 95% on clean digital invoices, which sounds like the problem is solved, until you look at scanned invoices and receipts, where accuracy drops back to roughly 87 to 93%. A single model update or an unfamiliar document variant can wipe out that gap overnight, with no warning unless something is watching for it.

What a regression test for document extraction needs to measure

Pass or fail is the wrong frame. Extraction regression is a signal that varies by field type, document class, and schema version all at once, and collapsing it into one number throws away exactly the information an engineer needs to fix anything.

Field-level precision and recall come first: which fields got extracted correctly, which got missed, which got invented. Character Error Rate, computed with Levenshtein distance, is still the closest thing OCR has to a gold standard. Clean printed text should sit under 1% CER; handwriting runs 3 to 5% even on a well-tuned system. Word Error Rate complements CER on prose-heavy fields where character-level noise matters less than whether the right words landed in the right order.

Tables need their own metric, because a system can extract every character correctly and still scramble which cell each one belongs to. Tree Edit Distance Similarity, or TEDS, checks whether rows, columns, and cell relationships survive the extraction as well as whether the text made it out. Normalized Information Distance is a compression-based similarity measure rooted in Kolmogorov complexity that compares extracted output to expected output without hand-built features. It's a useful general-purpose distance metric, but it wasn't built to catch paraphrase-level errors that slip past CER.

None of this matters if the metrics only get reported at the document level. A document can score 95% overall while a single critical field, line items or a nested address block, is 0%, buried under every field that extracted fine. Aggregate accuracy is a number that makes a broken system look healthy.

The array omission failure mode deserves its own callout, because it costs the most and gets caught the least. A parser that silently drops rows three and seven out of a twelve-row line-item table isn't obviously wrong. The document still "extracts." Aggregate accuracy metrics don't penalize this the way they should, which is part of the reason ExtractBench was built: to evaluate array alignment directly, and to draw a hard line between a field that's missing and a field that's fabricated. Schema breadth alone can break systems that look fine on smaller, simpler documents.

Building a ground-truth corpus from real production documents

Synthetic test documents can't replicate the ways real documents degrade. Scan quality varies by scanner, by the person who ran it, by how many times the paper got faxed. Templates drift. Someone writes a note in pen across a printed field. None of that appears in a corpus built from clean, generated samples, so the corpus has to come from production.

The best starting material is whatever already failed. Documents that triggered a human review, or that a customer flagged as wrong, are the hardest cases available and the most diagnostic ones, because they already proved the system's blind spot.

From there, stratify. Sort by document class, invoice type, form version, contract template, and by quality tier, clean digital, scanned, degraded scan. Within a single document type, keep multiple template variants: two invoices from the same vendor rarely look identical if the vendor's billing system changed once in the last three years. Build in the edge cases on purpose rather than waiting for one to appear in production: redlined text, strikethroughs, merged table cells, tables that span multiple pages, signature fields.

Annotation quality is the ceiling on everything downstream. Annotators need to label at field level, not document level, because a document-level label ("mostly correct") tells the regression suite nothing about which field broke. A regression suite is only as trustworthy as the ground truth it's checked against, and sloppy annotation guarantees a suite that either misses real regressions or flags phantom ones.

olmOCR's benchmark design is a useful reference point here because its structure is worth copying, even though it isn't a production tool. It splits its evaluation set into eight categories spanning conditions such as old scans with degraded ink, tiny text, mathematical notation, multi-column layouts, tables, and headers and footers. The underlying principle, deliberate diversity across document conditions rather than volume for its own sake, applies just as well to a corpus built from a company's own production traffic.

Structuring the regression suite: field-level tests, schema versioning, and diff taxonomy

Granularity is the whole game. One test per field, not one test per document. Coarser than that, and a regression on a single field hides inside a passing aggregate the same way it hid inside the accuracy score above.

Treat the schema itself as an executable specification. Each field declares its own scoring rule: exact match for identifiers like invoice numbers and tax IDs, a tolerance window for quantities like amounts and dates, semantic equivalence for names and free-text descriptions. Arrays need their own alignment logic, because a single row inserted at position three shouldn't cascade a failure through every row that follows it, which is exactly the trap a naive positional diff falls into. ExtractBench (arXiv:2602.12247) operates at benchmark scale on this principle, and the general approach ports directly into a production test suite without much modification.

Every regression also needs a category, not just a flag. Value drift means the right field carries the wrong value, a transposed digit, a truncated string. Structural drift means the field is there but nested wrong or sitting in the wrong position. Omission means the field never made it into the output. Hallucination means a field appears in the output with no basis anywhere in the source document. Reading-order drift means the text is correct but the sequence is scrambled, which matters enormously for anything that reads as prose. Each category points an engineer toward a different part of the pipeline, so lumping them together as "wrong" wastes the diagnostic value of running the test.

Schema versioning belongs in CI. Every schema change should trigger a full regression run against the current ground-truth corpus before that change gets promoted. That run is the gate. Skipping it lets schema drift happen silently, one small change at a time, until nobody can say which change broke which field.

Integrating regression testing into the deployment pipeline

Regression tests need to live in three different places, doing three different jobs. Pre-merge, run against a representative subset of the corpus, fast enough to catch the obvious breaks without slowing every commit to a crawl. Pre-deploy, run the full corpus against any model update, parser version bump, or schema change, because this is the last checkpoint before the change reaches real traffic. Post-deploy, run the new version in shadow mode alongside production, diff the outputs, and alert if divergence crosses a threshold before traffic shifts over completely.

Code changes aren't the only trigger. A model provider pushing a new version of an underlying LLM, a customer quietly switching to a new invoice template, a regulator updating a form, all of these can degrade extraction accuracy with zero lines of code touched. Regression runs need to fire on those events too. The pipeline has to watch for document format drift and provider updates, not just git commits.

The accuracy report generated at field and document level shouldn't be a dashboard someone glances at after the fact. It's a build artifact, and it should function as a gate: block the deploy, or pass it. A report nobody has to act on is a report that gets ignored the first time someone's in a hurry.

And the loop needs to close. Human corrections made on low-confidence outputs in production should feed back into the ground-truth corpus, expanding its coverage and making every future regression run a little harder to fool. Over time, that feedback loop is what keeps the corpus from calcifying around whatever failure modes existed the day it was built.

Where open-source benchmarks inform production regression, and where they fall short

OmniDocBench is a solid reference for methodology. It covers 981 samples drawn from a range of document sources, annotated at the layout, reading-order, and component level. That's a rigorous setup worth borrowing wholesale for internal eval design, even for teams that never touch the benchmark itself.

But benchmarks saturate, and when they do, they stop discriminating. Both GLM-OCR and PaddleOCR-VL-1.5 have crossed 94% accuracy on OmniDocBench, which is a strong result, but it also means the benchmark is approaching the point where a high score no longer tells you much about how a system handles the genuinely hard documents. A benchmark everyone's clearing isn't measuring the frontier anymore.

VAREX targets a different failure surface: 1,777 government forms carrying 1,771 unique schemas, essentially one schema per document. It surfaces failure modes that don't show up on more homogeneous benchmarks, most notably where a model produces output that conforms to the shape of the schema without actually pulling values from the document, an error that drags scores down substantially in the models it affects. VAREX also documents under-extraction and attention decay as distinct problems. What's encouraging is that extraction-specific fine-tuning on a model with a small number of parameters produced a substantial gain, which says the deficit isn't a scale problem so much as a training-target problem.

VAREX's ablation across four input modalities, plain text, layout-preserving text, document image, and text combined with image, is directly useful for anyone deciding how to feed documents into a parser. Layout-preserving text delivered the largest gain, a wide range depending on the case, beating pixel-level visual input. That's a concrete, actionable finding for teams choosing a parser's input representation.

Evaluation tooling options for production document extraction regression

Judge any tool against six things: how deep the measurement goes (field-level or just document-level), how flexible the metrics are (CER, TEDS, semantic similarity, custom scoring), how well its document coverage matches real-world variety, how cleanly it plugs into CI/CD, whether it supports a feedback loop back into the corpus, and how it handles security and compliance for sensitive documents.

OmniDocBench is open source through opendatalab and works well as adaptable academic infrastructure for an internal eval. An April 2026 update introduced MGAM to remove matching bias from its scoring, and a July 2026 update added a community-maintained EvalScope integration for OpenAI-compatible endpoints, producing standardized predictions, metrics, and reports. That's a real option for teams that want a standardized eval pipeline without building one from scratch.

olmOCR's evaluation protocol takes a stricter, binary approach: phrase presence, correct reading-order sequence, correct table cell column alignment, pass or fail on each. It's a hard test of factual correctness, and it deliberately doesn't penalize minor formatting differences. Best treated as a complement to a continuous-metric tool rather than a replacement for one, since binary pass/fail throws away the gradient information a team needs to track slow degradation.

ExtractBench, open source under arXiv:2602.12247 and arXiv:2607.29677, is the closest thing to a template for a production-style suite. Each test case pairs a document with a JSON Schema and expected output. The original release covers a large number of evaluatable fields, and the benchmark has expanded across subsequent releases. Its schema-as-executable-specification approach translates cleanly into finance, healthcare, logistics, or real estate, industries where the schema itself is often more complex and more idiosyncratic than the document text.

Operating a regression framework over time: corpus maintenance, alert design, and continuous improvement

A corpus built once and left alone starts decaying the moment a customer changes a template. Regulations update the forms they require. New document classes that nobody anticipated when the corpus was built appear in production. None of that is a one-time setup problem; it's ongoing curation, and treating it as finished is how a regression suite quietly stops meaning anything.

Three events should trigger a corpus update automatically. Any production failure the regression suite didn't catch belongs in the corpus immediately, because that's a gap in coverage that just proved itself real. Any new document variant, whether a customer rolled out a new invoice layout or a regulator changed a form, needs a slot in the corpus too. And any field that human reviewers keep correcting past some threshold in a rolling window is telling you exactly where the model is weak, in real time, for free.

Alerts need to separate real signal from noise, and that means at least three tiers. A hard block fires when a field drops below its threshold, and it stops the deploy outright, no exceptions. A soft alert fires on a degradation trend across multiple runs, even while the metric is still technically above threshold, because a slow slide is easier to fix before it crosses the line than after. An anomaly alert fires when low-confidence outputs spike in production on a field class the corpus doesn't even cover yet; this usually indicates an out-of-distribution document type just appeared and nobody built a test for it.

Regulatory context is starting to matter here too. The EU AI Act's requirements around high-risk AI systems put pressure on exactly this kind of documentation and monitoring, especially for financial and healthcare workflows where extraction errors carry real downstream consequences. A regression framework built the way this piece describes, field-level, versioned, actively maintained, is an engineering necessity at that point. It's the record that shows the system was actually being watched.