Retrieval‑augmented generation (RAG) is usually judged only by whether the final answer is correct. This criterion misses cases where the answer matches the reference but the retrieved context contains a direct contradiction, making the conflict invisible to answer‑only reviews or relevance scores. The paper introduces the Hierarchical Consistency Framework (HCF), a post‑hoc, model‑agnostic audit that examines three distinct stages of a RAG pipeline: the knowledge corpus, the retrieved context, and the generated answer. HCF models corpus conflicts as source‑linked atomic facts, identifies the documents responsible, and assigns an Answer Consistency Score (ACS) to each answer together with explanations of supporting and contradictory statements. Experiments were conducted on controlled corpora covering five domains, creating 100 query‑corpus instances. Human evaluators compared every generated response with its ground‑truth counterpart. Results show that the three diagnostic levels can dissociate: the corpus with the highest mean retrieval similarity often yields the lowest mean ACS, while a structurally degraded corpus performs poorly at the corpus level but better at the answer level. Most importantly, HCF uncovers contradictory retrieved evidence in cases where the answer still aligns with the ground truth. HCF does not certify factual truth; it makes the evidence for and against an answer inspectable and attributable.
Review