Affect-Salient Event Sequences Modelling for Continuous Speech Emotion Recognition Using Connectionist Temporal Classification

Continuous speech emotion recognition task faces the challenge of delays caused by reaction time, which is inherent in human annotations, and noises caused by non-emotional segments. To settle these, we propose an affect-salient event sequences modelling (ASESM) method based on connectionist temporal classification (CTC). The proposed method treats a sentence's label sequence as a chain of affect-salient event (ASE) states and Null (i.e., non-affect-salient event state), and models a CTC-based convolutional neural network (CNN) to automatically label the sentence's emotional segments with ASE and non-emotional segments with Null. Then, the continuous arousal and valence annotations of each ASE are used to mark the emotional value of the segment which is predicted as the ASE for testing samples. Our method avoids the reaction delay compensation by using events as the target and reduces the impact of noises by using CTC. Experimental results on the RECOLA dataset demonstrate the effectiveness of our method compared to state-of-the-art speech-only methods.