Emotion recognition from facial expressions and contactless heart rate using knowledge graph

The application of the knowledge graph in computer vision is a new trend in deep learning. Facial video-based emotion analysis and recognition are critical topics of research in the mental healthcare field. In this paper, we proposed a novel noncontact intelligent framework to represent the knowledge of facial features and heart rate (HR) features for predicting the emotional states of objects. The framework is divided into two parts: knowledge modeling and knowledge reasoning. In the first step of knowledge modeling, 3D-CNN is utilized to model the spatiotemporal information from the facial and forehead regions based on the remote photoplethysmography technique, separating the blood volume pulse (BVP) signal and extracting the HR from the forehead image sequence. Finally, the multichannel features are integrated and transformed into structured data and put into the knowledge graph as much as possible. Knowledge reasoning is an inferential process that associates the deep learning model with structured knowledge to predict continuous values of the emotional dimensions (pleasure, arousal, and dominance) from facial videos of subjects. Experiments conducted on the DEAP database demonstrate that this approach leads to improved emotion recognition performance and significantly outperforms recent state-of-the-art proposals. The result proved that prior knowledge from the knowledge graph ground truth on deep learning is an efficient means of emotional recognition in vision modality. Our artificial intelligence models can be popularized and applied in daily healthcare.