CharXiv is a benchmark designed to measure document understanding capabilities of large language models (LLMs) through character-level accuracy metrics. The benchmark evaluates how effectively AI systems can process, comprehend, and accurately reproduce textual content at the granular character level, providing a rigorous assessment of document comprehension performance across various document types and complexities.
CharXiv represents an advancement in LLM evaluation methodology, moving beyond traditional word-level or token-level accuracy measurements to assess character-level precision. This approach provides a more stringent evaluation framework that captures subtle differences in model outputs, including punctuation, whitespace, and formatting accuracy. Character-level accuracy serves as a comprehensive metric for evaluating document understanding, as it requires models to maintain fidelity across all textual elements rather than just semantic content 1).
The benchmark has emerged as an important metric within the rapidly evolving landscape of document processing and comprehension tasks for large language models, particularly for applications requiring high-fidelity text reproduction and understanding.
Character-level evaluation differs fundamentally from traditional token-based accuracy metrics commonly used in LLM benchmarking. While token-level accuracy groups characters into predefined vocabulary units, character-level accuracy assesses each individual character independently. This distinction becomes particularly significant when evaluating document understanding tasks where formatting, special characters, and exact textual reproduction are critical requirements.
The benchmark measures how precisely models can process documents and generate character-accurate outputs. This metric proves especially valuable for evaluating LLMs on tasks involving:
Notable performance results on CharXiv have demonstrated substantial capabilities in contemporary large language models. As of 2026, Kimi K2.6 achieves 86.7% character-level accuracy when processing documents using Python implementation 2), establishing a significant benchmark for document understanding at character-level precision.
This performance level indicates substantial progress in developing models capable of maintaining character-accurate comprehension across complex documents, suggesting advances in both model architecture and training methodologies for document processing tasks.
CharXiv benchmarking proves particularly relevant for applications requiring high-fidelity document processing, including:
Organizations deploying LLMs for document-intensive workflows benefit from character-level accuracy metrics, as these measurements directly correlate with the reliability and trustworthiness of model outputs in high-stakes document processing scenarios.
As document understanding has become an increasingly important capability for large language models, benchmarks like CharXiv provide quantifiable measurements for comparing model performance. Character-level accuracy offers a more nuanced evaluation than word-level metrics, capturing subtleties that matter in professional document processing contexts 3).
The emergence of character-level accuracy benchmarks reflects the maturation of the LLM evaluation landscape, where increasingly rigorous and domain-specific metrics guide model development and deployment decisions. As models continue to specialize in document processing tasks, character-level benchmarks serve as essential tools for assessing real-world applicability.