Research about feature fusion levels

[Luxshan2000]

Description: Investigate different levels of feature fusion (early, mid, and late fusion) in Speech Emotion Recognition (SER). Analyze their impact on performance and summarize findings with potential strategies for effective fusion.

[Jubeerathan]

Fusion Approach	Description	Key Methods	Notable Studies & Findings	Advantages	Disadvantages
Early Fusion	Fusion at feature level before model processing.	- Feature Concatenation - Model/Network Concatenation - Hierarchical Fusion	- Hazarika et al. (2018): Self-attention mechanism improved accuracy from 62.5% to 72.2% on IEMOCAP. - Atmaja et al. (2020): Improved valence prediction accuracy from 49% to 56.3% on USOMSe. - Majumder et al. (2018), Tian et al. (2016): Hierarchical fusion performed better than standard early/late fusion.	- Retains detailed multimodal information. - Can leverage deep networks like LSTMs, CNNs.	- High computational cost. - Risk of overfitting due to high-dimensional input.
Late Fusion	Fusion at decision level after independent processing of modalities.	- Majority Voting - Ensemble Learning - SVM-based Fusion	- Chen & Zhao (2020): Improved accuracy from 70.83% (bimodal concatenation) to 71.06% (decision fusion). - Cho et al. (2018): SVM fusion of acoustic, text, and e-vector outperformed unimodal approaches. - Atmaja et al. (2020): Improved CCC from 0.508 (early fusion) to 0.536 (late fusion).	- More robust to noise. - Reduces feature space complexity.	- Less interaction between modalities. - Loss of fine-grained cross-modal relationships.
Hybrid Fusion	Combination of early and late fusion.	- Mixed Feature-Decision Fusion	- Sebastian & Pierucci (2019): Used hybrid fusion on IEMOCAP dataset, but results were not significantly better than using a single approach.	- Potentially captures the best of both early and late fusion.	- Higher complexity without clear performance gain.
Performance Comparison	No consensus on best approach.	- Feature-level fusion found best by Planet & Iriondo (2012). - Atmaja & Akagi (2021) found opposite results. - Pepino et al. (2020) found no significant difference.	- Results vary based on dataset and method.	- Requires extensive experimentation to determine optimal fusion method.