Development and Testing of a Methodology for Evaluating the Effectiveness of Neural Network Models in Annotating Customer Helpline Calls

Abstract

The automatic speech recognition (ASR) problem is considered in the context of telephone calls from users to the support service for further processing and analysis of transcriptions. The issue of choosing a suitable neural network ASR model with transformer and conformer architectures among popular open source ASR projects is investigated. A methodology for testing ASR models based on a typical set of Russian-language recordings of real calls with taking into account configurable hyperparameters of the models has been developed. By applying the proposed methodology, several widely used ASR models were compared according to the recognition accuracy and performance criteria. Based on the comparison results, a model for the target telephone support service has been selected, and two assumptions have been formulated about the existence of a maximum amount of training data, after which the recognition accuracy depends mainly on the number of trained parameters. An approach for pipeline postprocessing is proposed, which includes speaker diarization and automatic annotation using large language models for subsequently forming a database of frequently asked questions and other applied tasks.