Neural Network Method for Emotion Speech State Recognition in Contact Centre Systems Based on CNN-BiLSTM Architecture with a Modified Attention Mechanism

Abstract

The relevance of this research lies in the need to improve the efficiency of decision-support systems in contact centers for automated analysis of the emotional states of operators and clients. Detecting emotional tension in the voice enables timely adjustments to interactions, enhancing service quality and operator performance. Traditional audio signal processing methods based on Mel-Frequency Cepstral Coefficients (MFCCs) have limitations in preserving complete acoustic information, thereby reducing the accuracy of emotion recognition. This work proposes a neural network method that combines Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTM) networks with a modified Attention mechanism. The first stage involves loading a Ukrainian-language audio dataset and performing preliminary data processing, including amplitude normalization, noise filtering, speech segmentation, conversion to Mel-spectrograms, and extraction of low-level descriptors (LLD), such as energy and fundamental frequency (F0). The input data are formed into fixed-dimension tensors for neural network analysis. During the feature extraction stage, the CNN automatically identifies local spectral characteristics of the signal, including intensity, frequency components, and intonational peaks. Each convolutional block is complemented with batch normalization to stabilize training and accelerate convergence. To model the temporal dynamics of the emotional state, bidirectional BiLSTM layers are applied, taking into account the context of preceding and subsequent signal segments. The Attention mechanism forms a context vector as a weighted sum of features, determining the relative importance of individual time frames, and then passes it to a fully connected layer with a tanh activation function. In this work, the modified Attention mechanism refers to the integration of audio recording metadata (signal duration, 640 kbps bitrate, technical identifiers) into the formation of the context vector via a Multi-Weighting System (MWS). It allows simultaneous consideration of local spectral features, temporal dynamics of the audio signal, and the relevance of individual speech segments. The scientific novelty lies in the development of a neural network method that combines CNN–BiLSTM–Attention with a multimodal weighting mechanism, integrating audio metadata into the formation of the context vector. This architecture provides increased accuracy in recognizing emotional tension. A comparative analysis of traditional MFCC and LLD was conducted, demonstrating the advantage of LLD: the baseline CNN accuracy increased from 82,42 % to 9,.00 %, and integrating the Attention layer further improved accuracy by 1.5...2%. The highest achieved accuracy was 93,48 %