Abstract
Recent advances in large language models (LLMs) and their extension to vision-language models (VLMs) have made it easier to combine text and images for tasks such as report generation. Existing VLMs in medicine typically focus on 2D images (chest X-rays), and their extension to 3D imaging has been difficult because of the lack of paired 3D imaging-text data. Thus, we introduce a new method for creating a 3D image-text dataset for brain oncology using 3D MRI scans of glioma and meningioma cases.
We use a cooperative system in which several LLMs work together to generate and check reports, ensuring that they are accurate and clear. By leveraging the new 3D MRI-text dataset, we further build a VLM that converts MRI scans into tokens and aligns them with text instructions. Our VLM performed better in report generation and visual question answering tasks than other 2D and 3D methods.
Our method not only improves the quality of reports but also helps with better diagnosis and treatment in brain oncology.
Blogger's Review: This paper demonstrates how to enhance the quality of medical report generation using a new dataset and multi-model collaboration, breaking through the limitations of traditional 2D image analysis, and providing a more effective tool for the diagnosis and treatment of brain tumors. Similar approaches are expected to be applied in more medical imaging fields in the future.