Abstract
Real-world document classification pipelines typically apply the same sequence of models to every incoming document, regardless of its complexity or type. This leads to inefficient use of compute and human resources: simple documents are over-processed while difficult ones may not receive enough scrutiny.
We introduce DocHRL, a hierarchical reinforcement learning framework that learns to adaptively and dynamically select the most cost-effective classification policy on a per-document basis. DocHRL formulates document classification as a sequential decision problem with a two-level policy hierarchy: a top-level policy selects among broad options (vision classifiers, LLMs, OCR, and human-in-the-loop review), while option-specific sub-policies choose the concrete model or tool to invoke.
The reward signal is the negative total expected cost, which captures inference cost, cost of misclassification, and cost of human labeling. Trained with Proximal Policy Optimization on the RVL-CDIP benchmark, DocHRL achieves a macro F1 of 0.973 across 16 document classes while reducing average per-document cost to 2.74 normalized units compared to substantially higher costs incurred by fixed standalone classifiers.
Our results demonstrate that cost-aware reinforcement learning can simultaneously improve classification performance and operational efficiency in document understanding systems.
Blogger's Review: The innovation of DocHRL lies in its hierarchical strategy selection mechanism, leveraging the strengths of reinforcement learning to enhance both efficiency and cost-effectiveness in document classification. This framework not only focuses on accuracy but also incorporates cost control, showcasing significant potential in practical applications.