In specialized fields such as medical diagnostics and financial advisory, the deployment of large language models (LLMs) requires evaluating capabilities beyond general knowledge. Traditional question-answering benchmarks often fail to capture the nuanced contextual grounding, user awareness, and domain understanding needed in these fields. To address this, we introduce CANDI-QA (Contextual Alignment for Niche Domains Question Answering), a novel dataset that evaluates LLMs on delivering accurate, context-sensitive, and user-aligned answers in specialized settings. CANDI-QA features expert-curated question-answer pairs structured into two categories:
- Information Assistance Questions: direct factual queries requiring precise extraction.
- Applied Inference Questions: multi-hop reasoning tasks needing situational inference to generate actionable insights.
We evaluate over ten diverse language models, from compact open-source to state-of-the-art proprietary systems. As a robust baseline, we present MTSS-Net, a lightweight neuro-symbolic framework combining neural retrieval with rule-based reasoning. Our findings highlight the profound challenges of achieving contextual alignment in niche domains, revealing the limitations of current LLMs without enhanced contextual or symbolic integration. Ultimately, CANDI-QA serves as a critical benchmark for advancing research in context-aware language models, stimulating the development of robust, trustworthy AI for high-stakes domains.
Blogger's Review: The introduction of the CANDI-QA dataset provides a crucial evaluation standard for question-answering systems in niche domains, especially given the challenges LLMs face regarding contextual alignment and reasoning capabilities. Future models need to better integrate contextual information with symbolic reasoning to enhance performance in specialized fields.