Document parsing is a foundational step for document understanding tasks such as visual question answering and key information extraction, transforming unstructured scanned images into structured representations by extracting textual, visual, and layout information. Despite the development of numerous Optical Character Recognition (OCR) engines and multimodal large language models (MLLMs), selecting the appropriate document parsing solution for a given document collection remains challenging, especially in label-scarce settings.
In this work, we conduct a systematic evaluation of text recognition performance across various OCR engines and state-of-the-art MLLMs on multiple scanned document benchmarks spanning different domains and languages. Motivated by the limited contextual reasoning capabilities of many OCR engines and the high cost of manual annotations, we propose DocOCR-Eval, an annotation-free evaluation framework for automatic OCR assessment and selection. DocOCR-Eval employs a three-staged correction and ranking strategy to approximate annotation-based tool ordering without ground-truth labels.
Our findings show that aggregating across multiple MLLMs progressively improves alignment with annotation-based rankings. Extensive experiments further demonstrate that reliable OCR tool selection can be achieved in realistic, label-limited settings, providing practical guidance for deploying document parsing systems across diverse real-world document collections.
Blogger's Review: The DocOCR-Eval framework offers an effective strategy for OCR tool selection without the need for annotations, holding significant practical value, especially in resource-constrained scenarios. Its three-stage correction and ranking strategy brings new insights to the document parsing field, advancing the application and development of OCR technology.