NeFut Logo NeFut
中 Admin Login

[CS.AI] BrainBind: Bridging Synchronized EEG and fMRI through Long-Form Naturalistic Video Representations

Published at: 2026-11-05 22:00 Last updated: 2026-11-07 12:32
#AI #Machine Learning #Neural

Real‑world vision is dynamic and scene‑rich, while EEG and fMRI complement each other in temporal and spatial resolution. Most neural decoding works align a single modality, often using static images or short clips. Here we study the synchronized case: EEG and fMRI are recorded while subjects watch the same long‑form naturalistic video, providing a shared stimulus together with recurring narrative context and many semantically similar candidates.\n\nThe BrainBind framework projects synchronized EEG and fMRI into frozen long‑form video embedding spaces and treats these embeddings as common visual‑semantic references rather than literal neural codes.\n\nEvaluation on the CineBrain dataset uses a five‑seed Recall@K matrix (2×4) covering two video anchors and four system configurations, revealing distinct retrieval profiles. The Omni configuration organizes the full candidate list most strongly, achieving $MRR=23.82$ and $nDCG@10=25.31$, while Joint‑Dual attains the highest label‑aware $R@1$ (13.52%) with a unique rank‑wide pattern.\n\nAn episode‑disjoint evaluation quantifies how repeated stimulus identity and narrative context contribute to random splits. Across experiments, proxy choice and system configuration generate different top‑rank and rank‑wide retrieval characteristics within the synchronized dataset. The code is released at https://github.com/Forainest789/Brainbind.\n\n**Review**: BrainBind’s approach of mapping neural signals to a unified visual‑semantic space offers a fresh perspective for cross‑modal retrieval, especially under long‑form natural video conditions where it demonstrates solid performance. Future work could explore finer temporal alignment and richer multimodal fusion strategies.

Original Source: https://arxiv.org/abs/2611.05581

[h] Back to Home