Multimodal large language models (MLLMs) are advancing quickly, yet conventional benchmark development lags, causing delays in identifying newly observed capability gaps. Chart question answering (Chart QA) is well suited for probing coupled perception and reasoning because charts are information‑rich. Automated Chart QA construction aims to shorten the benchmark cycle by turning identified gaps into targeted diagnostic samples on demand. Existing approaches usually separate target guidance from scratch generation: target‑guided systems need pre‑prepared data, charts, or templates, while scratch‑generation systems focus on artifact validity without explicit control over alignment with an external diagnostic target. ChartBmkAgent introduces a central harness that orchestrates specialized agents, requires stage‑specific evidence of alignment with the original error category, and logs the rationale for each acceptance decision. On 300 taxonomy‑wide samples, MLLM accuracies ranged from 32.7% to 84.3%, revealing distinct category profiles and indicating that the generated samples expose capability differences. Comparisons across three source models showed targeted follow‑ups scoring 50.0% versus 82.2% for matched controls ($p=8.96\times10^{-6}$); all six cross‑model comparisons shared the same direction, demonstrating the effectiveness of targeted validation and diagnostic data generation. Six evaluator models assessed whether each sample tested its specified error category, with 86.4% meeting this criterion, providing empirical evidence of target preservation.
Review