Abstract
LLM-as-a-judge is widely used to provide feedback and selection signals in closed-loop regeneration, but this use remains insufficiently validated. We study it in table recognition, where deterministic TEDS evaluation provides a controlled testbed, using FinTabNet and OmniDocBench.
Key Findings
-
Weakness of Judge Signals: Judge signals were weak on both datasets: scores frequently tied, rankings were not reproducible, and the only selection policy that beat random on both datasets depended on an earliest-iteration tie rule, so its advantage cannot be attributed to the judge scores alone. Iteration produced better candidates, but the judge failed to recover them.
-
Severe Losses Occurred: Severe losses occurred even without specific judge feedback. A structure-preserving instruction significantly reduced the severe-loss rate on FinTabNet and was directionally consistent on OmniDocBench. The contrasts support target-preservation failure under unconstrained regeneration as a proximate mechanism of the observed severe losses.
-
Impact of Structure-Preservation Constraint: The structure-preservation constraint reduced the severe-loss tail but produced no improvement. In an exploratory 2x2 analysis, the same protection was not stably observed when judge feedback was retained.
These results do not dispute the value of LLMs as evaluators. Instead, they show that evaluation ability does not imply optimization utility. Iterative refinement requires, at minimum, a verification signal that deterministically detects structural change, rather than judge scores alone.
Blogger's Review: This study reveals the limitations of LLMs in closed-loop systems, highlighting the gap between evaluation and optimization. While LLMs can provide feedback, relying solely on judge scores may not suffice for effective optimization. This points future research toward the need for more effective verification mechanisms.