NeFut Logo NeFut
中 Admin Login

[CS.AI] Target-Aware Curation of Hallucination Stress Tests for Large Language Models

Published at: 2026-10-23 22:00 Last updated: 2026-10-25 13:04
#Machine Learning #LLM #Artificial Intelligence

Hallucination stress tests often yield comparable average scores across language models, yet different prompts can trigger hallucinations uniquely in each model. We introduce an iterative procedure that edits source‑grounded seed prompts to expose hallucinations in a target model. In each round, an LLM editor revises the prompt based on the target's answer and a judge's category‑specific score and explanation; in the main setting the target also receives feedback on its previous response. Experiments cover 500 seeds drawn from seven datasets and ten topics, evaluating intrinsic and extrinsic hallucinations on both open‑weight and closed‑source models. Editing substantially raises the judge‑rated severity of hallucinations. In the baseline configuration, retained responses fall into the strong‑hallucination bin in 77.2% of runs targeting extrinsic hallucination and 74.2% targeting intrinsic hallucination. Later editing rounds account for more than half of the mean score increase. Among 219 prompts that pass model‑assisted source‑validity checks, 89 elicit unsupported claims. Across five models, 200 curated prompts yield similar average severity scores, yet pairwise Spearman correlations between prompt‑level scores range only from $\rho=0.0387$ to $0.1897$. Curated prompts can be reused across models, but their diagnostic value depends on source validity and the specific failures they expose in each model.

Review

Original Source: https://arxiv.org/abs/2610.27247

[h] Back to Home