The complementary information between RGB and IR images significantly enhances object detection performance under extreme conditions. Existing methods typically rely on dual-stream CNN backbones built on YOLO for feature extraction, focusing primarily on feature fusion design. This paper introduces the Laplacian Decoupled Feature Enhancement block (LDFE) to fuse features from different stages of the dual-stream CNN backbone. LDFE employs a sequence of global-local decomposition, denoising, fusion, and reconstruction, considering the characteristics of modalities and structures for feature fusion.
The LDFE first separates features into global and local components using the Laplacian Pyramid, followed by denoising and fusion through the Global State Space Enhancement module (GS2E) and Local Convolutional Correlation Enhancement module (LC2E). Specifically, GS2E implements a two-branch architecture for the main and auxiliary modalities, dynamically suppressing noise in the main modality via cross-modal attention from the auxiliary modality while employing a State Space Model to capture long-range dependencies in the global feature representations.
The two modalities systematically alternate their main/auxiliary roles to obtain bidirectional interaction. Moreover, LC2E suppresses noise in local features and utilizes spatial and channel dimensions along with triple convolution to extract fine-grained details for fusion. These innovative designs yield significant performance improvements, with mAP exceeding SOTA methods by 6.2%, 3.7%, 4.7%, 2.3%, 4.1%, and 2.0% on the M3FD, DroneVehicle, LLVIP, FLIR-Aligned, KAIST, and VEDAI datasets, respectively.
Blogger's Review: The LDFE module, with its innovative global-local feature processing approach, significantly enhances the fusion of RGB and IR images, providing robust technical support for object detection in extreme environments. Its outstanding performance across multiple datasets further validates the effectiveness and application prospects of this method.