Analyzing merchant transaction histories requires dynamic temporal filtering, entity resolution, and aggregation over irregular event streams—operations that static tabular or time‑series forecasting benchmarks fail to capture. We introduce TxnQA, an analytical QA benchmark comprising 27 task families (81 templates) together with a 5‑scenario fraud‑risk reasoning suite that evaluates both verdicts and supporting evidence on synthetic, privacy‑mitigated transaction data. Ground‑truth answers are generated programmatically, and model performance is broken down into format‑following rate (FFR), conditional content accuracy (CA), and end‑to‑end correctness (E2E). Comparing prompted proprietary baselines with six LoRA‑adapted open‑weight models shows that adaptation reliably enforces output schemas, yet arithmetic reasoning remains a bottleneck: Gemma‑4‑31B‑IT and Qwen3‑8B achieve E2E scores of 0.867 and 0.816, while evaluated MoE checkpoints range only between 0.632‑0.713. Moreover, on the 500‑instance fraud suite, verdict accuracy often exceeds 80%, but joint accuracy (requiring exact evidence attribution) collapses to at most ~23%. These findings identify schema compliance, semantic calculation, and evidence attribution as three distinct bottlenecks in transaction‑sequence QA.
Review