As frontier AI systems are increasingly used as sources of political and current‑events information, assessing the epistemic quality of their responses has become critical. Existing evaluations often focus narrowly on ideological alignment, rely on metrics not validated for this setting, or depend on reference labels derived from public opinion or non‑expert annotation, which can produce misleading or miscalibrated results. To overcome these limitations, we introduce NewsBench, an expert‑grounded benchmark that reframes evaluation around editorial standards developed by senior practitioners in journalism, policy, and intelligence analysis. NewsBench operationalizes quality across three dimensions—source quality, factuality, and neutrality—using a two‑phase pipeline: domain experts first define and calibrate evaluation guidelines; these guidelines are then transferred to automated AI judges validated against expert‑adjudicated gold labels. The benchmark includes roughly $3,000$ evaluation prompts, with about $500$ refreshed monthly to capture emerging events, roughly $2,500$ human‑expert‑annotated items for calibration and validation, and a suite of AI judges enabling scalable evaluation. We report inter‑rater reliability among human experts and AI judge–human agreement across all three dimensions to validate the methodology. Finally, we apply NewsBench to four frontier models, revealing distinct and non‑overlapping failure profiles in sourcing, factuality, and neutrality.
Review