We describe the SciTrue team's approach to both subtasks of the NTCIR‑19 SciClaimEval task, which requires systems to verify scientific claims against a paper's tables and figures. Instead of fine‑tuning a single model, we applied an honest per‑sample protocol to benchmark eleven frontier and open multimodal models and combined them with light, transparent post‑processing.
On the official blind test leaderboard, SciTrue ranked first by a clear margin in three of the four evidence‑category/subtask combinations and tied for first on the primary metric in the fourth. Three findings explain these results. First, strong instruction‑tuned models are already competitive: Claude Opus 4.8 and Gemma‑4‑31B both surpass the strongest public baseline (o4‑mini), while GPT‑5.5 and Claude Fable 5 lead both subtasks (Subtask 2 reaches 97.7). Second, the task's pairing structure provides the largest lever: a leak‑free pair prior recovers the Supported/Refuted pairing from the claim text alone (a visible field) and assigns Supported to the higher‑confidence evidence, raising Subtask‑1 pair‑accuracy from 72.2 to 93.5—far more than any model swap or ensemble weighting. Third, a case‑by‑case audit shows that most residual errors are visually undetectable label‑mapping swaps or dataset label noise, so measured accuracy understates true ability and the fixable‑by‑modeling headroom is small. Controlled fine‑tuning, distillation, and agentic consistency‑checking support the same conclusions; we also document a measurement leak—label information reaching a system through data packaging rather than content—where the released file ordering encodes the label, briefly misleading our pipeline.
Review: This work demonstrates that, in scientific claim verification, the synergy of multi‑model benchmarking and a pairing prior can dramatically boost performance, while also highlighting the need to guard against subtle data‑level leaks.