Abstract
•An automatic audio-visual emotion recognition system based on CNN was proposed.•The audio-visual features were fused using a cascaded deep sparse autoencoder.•Signals were distributed to different cloud servers using multi-secret sharing scheme.•Edge computing was introduced to minimize the burden of transmission.
This paper proposes an edge-cloud based privacy-preserved automatic emotion recognition system in which Internet of Things (IoT) devices capture facial image and speech signals from a user before distribution to different edge clouds using a secret sharing scheme, ensuring the privacy of the user. Simple signal preprocessing is performed in the edge clouds and then the signals are transmitted to the core cloud. In the core cloud, a Convolutional Neural Network (CNN)-based, pre-trained model is applied to extract deep-learned features from image and speech signals. Following the CNN model, two deep sparse auto-encoders fuse the signal features and introduce an extended non-linearity to the features. Finally, a support vector machine classifies the input in a corresponding emotion class. Several experiments were performed using the proposed system on two publicly available databases, namely, the RML database and eNTERFACE’05 database. The highest recognition accuracies by the proposed system on the two databases were 82.3% and 87.6%, respectively. These accuracies are higher than those by other state-of-the-art systems.