Multimodal large language models (LLMs) are increasingly deployed for structured extraction from financial documents, yet their behavior under visual ambiguity remains poorly understood. This work investigates the CARLAR (Courtesy Amount Recognition and Legal Amount Recognition) phenomenon: a model outputs the same amount for both cheque fields even when their annotations differ, including cases where a blank annotation is encoded as zero.
We evaluate GPT‑4.1 and Claude 4.5 Sonnet across ten visual prompting conditions on 243 primary cheques (124 with observed unequal amounts) and 598 strict mismatches from two additional datasets. Under courtesy redaction, GPT‑4.1 exhibits a CARLAR rate of 73.7% (95% bootstrap interval: 68.3‑79.0; p<0.001 vs. baseline). Model‑reported confidence stays high when predictions agree, and the expected calibration error (ECE) reaches 0.93 for CARLAR cases.
We also test a compressed‑image few‑shot intervention that presents examples as a single visual table without fine‑tuning. On the 243‑case cohort, Claude’s control condition improves paired joint extraction accuracy by 11.5 percentage points (95% interval: 6.2‑16.9; adjusted p=0.00054), though the gain on the strict‑124 subset is smaller and not significant after correction. The intervention’s effect on predicted‑field agreement is model‑dependent: it decreases for Claude but increases for GPT‑4.1.
These findings motivate validation methods that explicitly account for visual context and inter‑field dependencies in multimodal structured extraction tasks.
Review