Accurate Identification of Human Emotional States from Images Using Deep Learning

Emmy Yang; Jake Y Chen

Abstract

Facial expression recognition is a crucial aspect of human communication, especially for building social relationships. However, machine-based recognition remains a challenging task. Our research proposes an automatic emotion identification system that utilizes emotional state heatmaps (ES-MAPs) and neural network classification algorithms. Using MediaPipe Face Mesh, our system extracts facial landmark coordinates and calculates the distance between all landmarks. A neutral baseline is subtracted from the landmark distances and saved as a heatmap to train a designed CNN model. Our proposed system, ESH-Net, achieved significantly higher test accuracies on several datasets compared to other SOTA models. In addition, ES-MAPs produced better clustering than the original facial images, indicating significant improvement in the separability and consistency of representation of emotional states. This study demonstrates the potential for emotional state heatmaps and deep learning models to significantly improve the accuracy and efficiency of emotion identification, which can greatly assist in assessing patient's emotional state in medical diagnosis and practice.

Keywords: Facial Emotion Recognition; Deep Learning; Machine Learning; CNN; Neural Network

Links
[Full text PDF][Bibtex]