Using an expanded set of 722 official UN Security Council records, this work defines five complementary tests to measure nation‑level bias in large language models (LLMs): opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded evaluation, each test is assigned an explicit unbiased status; for WOW and AK+-, where a true ground truth is elusive, an action‑based pseudo‑unbiased status derived from observable voting behavior is introduced as a quantitative baseline.
Four models—GPT‑4o‑mini, Llama 3.3‑70B, Mistral‑22B‑Small, and Qwen3‑30B—are benchmarked across the five permanent members of the UNSC. Across all tests, a consistent macro‑pattern emerges: model outputs lean toward the opposition (red) side for China and Russia, and toward the supportive (blue) side for the U.K., the U.S., and France.
The results underscore the necessity of auditing IR‑focused LLM deployments with multi‑test benchmarks anchored to explicit reference points, thereby mitigating operational risks from systematic nation‑level bias.
Review: The paper delivers a systematic, quantifiable bias assessment framework and innovatively addresses the lack of ground truth for certain tests by leveraging voting behavior, offering a reproducible approach for future model audits.