ParseBench is a specialized benchmark designed to evaluate the faithfulness and accuracy of optical character recognition (OCR) and document retrieval systems, with particular emphasis on their reliability for agent-based applications. The benchmark comprises over 167,000 rule-based tests that systematically assess the quality of extracted document content across multiple dimensions of correctness and fidelity ¹⁾.

ParseBench addresses a critical gap in document processing evaluation by focusing specifically on content faithfulness—ensuring that extracted text accurately represents the original document without omissions, fabrications, or structural misinterpretations. This focus is particularly important for autonomous agents that rely on document understanding to make decisions, retrieve information, or generate outputs based on source material. Traditional OCR benchmarks may measure character-level accuracy without capturing whether agents receive semantically complete and accurately ordered information ²⁾.

The benchmark serves as a validation framework for document processing pipelines that feed into larger AI systems, ensuring that downstream agents work with reliable source material rather than corrupted or hallucinated content.

ParseBench systematically evaluates document processing through three primary dimensions of content fidelity:

Omissions refer to missing content from the extracted document—situations where the OCR or retrieval system fails to capture text that exists in the original source. This directly impacts agent performance, as agents operating with incomplete information may make incorrect decisions or provide incomplete responses to users.

Hallucinations occur when the document processing system introduces content that does not exist in the original document. This presents a distinct challenge from omissions, as fabricated text can mislead agents into generating incorrect outputs with high confidence, potentially amplifying errors through downstream decision-making.

Reading-Order Violations assess whether extracted content maintains the correct spatial and logical sequence as it appears in the original document. For agents that reconstruct narratives, follow procedural instructions, or rely on document structure, incorrect reading order can fundamentally alter meaning and lead to incorrect operations ³⁾.

With over 167,000 rule-based test cases, ParseBench provides comprehensive coverage of document processing scenarios. The rule-based approach allows systematic generation of test cases that target specific failure modes rather than relying solely on empirical examples. This enables more thorough evaluation of edge cases and systematic weaknesses in document extraction systems.

The benchmark's scale permits statistically robust assessment of system performance across diverse document types, layouts, and content configurations. Rule-based testing also facilitates transparent evaluation criteria, where specific rules define what constitutes correct extraction, making the benchmark's methodology reproducible and verifiable.

ParseBench's emphasis on agent-appropriate evaluation makes it particularly relevant for systems that combine document processing with autonomous decision-making. As AI agents increasingly operate over document collections—whether in research, legal discovery, financial analysis, or knowledge work—the reliability of extracted content becomes critical infrastructure. Agents that receive hallucinated or omitted information propagate these errors through reasoning chains and action sequences, potentially causing cascading failures in downstream processes.

The benchmark enables developers and organizations to quantify the trustworthiness of their document processing pipelines before integrating them into agent systems, reducing the risk of deploying systems that operate on corrupted source material ⁴⁾.

• Optical Character Recognition (OCR)
• Agent-Centric OCR
• CharXiv