Abstract
EEG-to-image evaluation should distinguish visual fidelity from recoverable meaning. Yet EEG-derived reconstructions are blurry, distorted, and low-detail, causing SSIM, LPIPS, and CLIP to penalize semantically recoverable outputs or reward plausible but incorrect ones.
We analyze 6,855 ground-truth/reconstruction pairs from ATM, ENIGMA, BrainVis, and DreamDiffusion using semantic probes, caption harshness, blind-spot rates, and controlled degradations.
Pixel metrics show near-zero correlation with semantic consistency, while representation metrics conflate perceptual and semantic errors. Therefore, we introduce a BCI-aware framework in which four VLMs assess image pairs through structured questions, producing Tolerant Perceptual Alignment Scores (T-PAS) and Tolerant Semantic Alignment Scores (T-SAS).
Their consensus is distilled into the BCI-Coherence Score (BCS), a compact evaluator achieving a T-PAS MAE of 0.079 (r = 0.700) and a T-SAS MAE of 0.082 (r = 0.850) on our data. Human validation shows highly reliable joint coherence judgments, with Cohen's kappa = 0.882 ± 0.174 and Krippendorff's alpha = 0.882, supporting perceptual-semantic recoverability over generic visual similarity. Code and resources are available at BCS GitHub.
Blogger's Review: This study introduces an innovative framework leveraging VLMs to enhance the quality of EEG-to-image reconstruction, emphasizing the importance of semantic consistency in visual quality assessment. This provides a significant theoretical foundation and practical guidance for future brain-computer interface applications.