NeFut Logo NeFut
Admin Login

[CS.AI] Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

Published at: 2026-08-13 22:00 Last updated: 2026-08-14 00:05
#Machine Learning #LLM #Open Source #DeepSeek #GPT

A recent study proposes a novel evaluation framework, Q-CARE, to assess the retrieval-augmented generation (RAG) capability of large language models. This framework enables fine-grained analysis of queries and answers by decomposing them into sub-queries and atomic claims, respectively. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). The experimental results demonstrate that Q-CARE outperforms four existing RAG evaluation metrics on a human-annotated benchmark spanning eight datasets. The code and data are publicly available at https://github.com/DISL-Lab/Q-CaRE-COLM-26. Blogger's Review: The Q-CARE framework provides a new perspective on evaluating RAG models by analyzing queries and answers at a fine-grained level. This framework can better assess the retrieval and generation capabilities of large language models, which is crucial for improving their reliability and practicality.

Original Source: https://arxiv.org/abs/2608.11238

[h] Back to Home