Large Language Models (LLMs) tend to hallucinate when answering domain-specific questions from scientific documents without prior fine-tuning. Existing methods like Retrieval-Augmented Generation (RAG) partially address this issue but face challenges, including limited contextual knowledge, differences between sparse and dense retrieval, and retrieval noise. This paper presents an Advanced Multimodal Retrieval-Augmented Generation (RAG) system aimed at overcoming these challenges and improving information extraction accuracy.
The proposed architecture features a multimodal ingestion pipeline that utilizes an open-source Vision-Language Model (Qwen2-VL-2B-Instruct) for generating textual summaries of tables and figures. The retrieval phase integrates HNSW-based semantic search with GIN-based lexical search, unified through Reciprocal Rank Fusion and refined with Cross Encoder reranking to minimize retrieval noise. A Query Condenser module is employed to ensure conversational coherence across multi-turn interactions.
Evaluation is conducted by independently assessing the ingestion, retrieval, and generation stages using the MMLongBench benchmark, a BeIR-format synthetic dataset, and the DeepEval framework. Results show a 157% improvement in retrieval quality over a Naive-RAG baseline with only 50 ms additional latency, while Qwen2-VL-2B-Instruct achieved results comparable to cloud-based models in BERTScore. These findings validate that open-source optimized SLMs, combined with advanced retrieval strategies, can provide competitive performance for document understanding without relying on cloud-based models.
Blogger's Review: This paper showcases an innovative multimodal retrieval-augmented generation system that significantly enhances the capability of scientific document understanding. By integrating various retrieval strategies with open-source models, the system excels in handling complex information, paving the way for broader applications in the research domain. Notably, the reduced reliance on cloud services will encourage the development of more localized solutions.