LLM decision components that operate within agent workflows often emit action‑relevant recommendations together with explanations. Operators rely on these explanations to monitor systems, diagnose errors, or decide when to escalate an output, implicitly assuming that the explanations align with the component's observable decision behaviour. This work examines two interpretations of the named factors in an explanation: necessity—changing the factor would change the output; sufficiency—retaining the factor while removing other changeable information would preserve the output. We evaluate both interpretations in two synthetic use cases: recommending advisors to clients and judging prompts for harmfulness or risk. Each model returns an output and the top three factors that most influenced it. Controlled black‑box interventions estimate a necessity score (how often altering the factor changes the output) and a sufficiency score (how often keeping the factor while discarding other information preserves the output). Across eight models from the Claude, GPT, and Gemini families, the mean Spearman correlations between the cited ranking and the necessity and sufficiency scores are 0.349 and 0.354 for advisor recommendation, and 0.431 and 0.580 for prompt monitoring. Moreover, an uncited factor scores above the lowest‑scoring cited factor in 57.6% of advisor responses under necessity and 58.1% under sufficiency; the corresponding rates for prompt monitoring are 25.8% and 8.9%. The cited top three contain useful information but do not reliably identify the three factors with the strongest measured influence under either metric. The framework offers a black‑box reliability check for explanations used in agent oversight while remaining scoped to individual LLM decisions.
Review: The study provides a practical method to quantify explanation trustworthiness through behavioural interventions, offering empirical support for explanation use in real‑world systems, yet further work is needed to handle more complex tasks and finer‑grained factor analysis.