Abstract
Real-time recognition of driver emotions can greatly improve traffic safety. With the rapid development of communication technology, it becomes possible to process large amounts of video data and identify the driver's emotions in real time. To effectively recognize driver's emotions, this paper proposes a new deep learning framework called Convolution Bidirectional Long Short-term Memory Neural Network (CBLNN). This method predicts the driver's emotion based on the geometric features extracted from facial skin information and the heart rate extracted from changes in RGB components. The facial geometry features obtained by using Convolutional Neural Network (CNN) are intermediate variables for the heart rate analysis of Bidirectional Long Short Term Memory (Bi-LSTM). Subsequently, the output of Bi-LSTM is used as input to the CNN module to extract the hear rate features. CBLNN uses Multi-modal factorized bilinear pooling (MFB) to fuse the extracted information and classifies it into five common emotions: happiness, anger, sadness, fear and neutrality. Our emotion recognition method was tested, proving that it can be used to quickly and steadily recognize emotions in real time.