Artificial Intelligence is increasingly embedded in complex sociotechnical systems, including Critical National Infrastructure (CNI). Harms typically arise from interactions among technical, human, and organisational elements, yet current AI evaluation remains model‑centric and offers little insight into how observed behaviours translate into system‑level risk. To bridge this gap we propose a framework that integrates structured hazard analysis, component‑level testing, and probabilistic system modelling, establishing a traceable pathway from model behaviour to system outcomes. The framework enables practitioners to answer the "so what?" of AI failures, quantify their systemic impact, and move toward evidence‑based, anticipatory AI governance. As a worked example we apply the approach to the UK Real‑Time Gross Settlement (RTGS) system, deriving AI‑driven loss scenarios with Systems Theoretic Process Analysis (STPA) and focusing on adversarial manipulation of an LLM‑based trading module. Component experiments show that simple adversarial inputs cause measurable shifts in AI recommendations, and when mapped onto a financial contagion model these shifts reduce system resilience, increase bank failure rates, and lower the shock threshold that triggers cascading disruption, especially under widespread or monopolistic AI adoption.
Blogger's Review: This study offers a comprehensive link from micro‑level AI behaviour to macro‑level system risk, providing valuable guidance for AI safety governance in critical sectors such as finance.