Ensemble Modeling for Emotion Recognition Using a Frame Attention Network on Faces With and Without Mouth Occlusion

Abstract

Facial Expression Recognition (FER) often struggles with partial occlusions, such as when a mask is present. Expressing emotion on the face while speaking is another challenge, as mouth movement conveys both words and emotion. FER models trained without these variations often fail to correctly classify unobserved patterns. Instead of (i) building a single model to learn these variations from the beginning or (ii) training three models, each specifically designed for one scenario (without occlusion, with mouth occlusion, and while speaking without occlusion), we create two models based on a frame attention network with two different datasets: one without any occlusion and the other with mouth occlusion. We then ensemble these models in the FER system, using the average of the scores from both models as the predicted expression. Experiments on the Ck+, Oulu-Casia, and Revdess datasets show that our proposed ensemble model, when tested on samples with no occlusion (regardless of whether the subject is speaking or not) and with mouth occlusion, achieved 52.83% accuracy, outperforming models trained specifically for faces without any occlusion and faces with mouth occlusion by 4.33% and 1.95%, respectively. It can also achieve approximately the same accuracy (0.05% higher) as the model trained on the dataset containing all variations, while saving the time required for retraining the model.