Cross-subject Multimodal Emotion Recognition via Resting-state EEG Adaptation
Electroencephalography (EEG) provides a direct, temporally precise window into affective brain dynamics, mak ing it a valuable modality for emotion recognition. However, cross-subject EEG-based emotion recognition faces persistent challenges due to substantial inter-subject variability and the scarcity of labeled data from new users. While multimodal fusion and deep learning have improved performance in con trolled settings, their generalization across individuals remains limited. To address this, we propose ResMF, a novel transfer learning framework that personalizes a general model using only unlabeled resting-state EEG, eliminating the need for task specific calibration. ResMF disentangles subject- and emotion specific representations and employs a Cross-Modal Attention mechanism to dynamically fuse EEG signals with facial Action Unit features. Experiments on two datasets demonstrate that ResMF consistently outperforms representative baselines. Our findings suggest that leveraging resting-state EEG alongside multimodal cues holds strong potential for building more robust and scalable emotion recognition systems. Index Terms—EEG, emotion recognition, multimodal fusion, generalization, unsupervised domain adaptation.