Recent advancements in Multimodal Large Language Models (MLLMs) have driven the development of end-to-end MLLMs for autonomous driving. However, the focus has primarily been on MLLMs utilizing 2D images and videos. This paper, in contrast, investigates the effectiveness of MLLMs with 3D sensors, particularly LiDAR and stereo cameras. LiDAR presents unique challenges for integration into an MLLM due to data sparsity and the absence of grid structures. Consequently, the fusion of camera and LiDAR data within an MLLM pipeline is uncommon. Nonetheless, most autonomous systems depend on LiDAR-based sensing, and incorporating 3D data has been shown to enhance performance in traditional 3D scene perception tasks.
This paper introduces D3VL, a novel MLLM framework that integrates 2D and 3D time-series data within a single, streamlined architecture. The model aims to address questions related to traffic scene understanding and safety. D3VL demonstrates an 11% improvement on the KITTI Question-Answering (QA) dataset compared to baseline methods while processing 2D and 3D time-series data. Additionally, the paper introduces the Waymo QA dataset extension, assessing models' capabilities in handling 3D and time-series data across diverse driving conditions. The implementation code for D3VL and the WaymoQA extension is available on our supplemental website: https://automotivesafety-lvlm.github.io.
Blogger's Review: The introduction of the D3VL framework signifies a pivotal shift in the emphasis on 3D data processing within the autonomous driving sector, particularly the innovative fusion of LiDAR data. By effectively integrating 2D and 3D time-series data, D3VL not only boosts performance in question-answering systems but also offers new insights for future developments in autonomous driving technology. Its application potential across diverse driving scenarios is noteworthy.