The Classification Quality Assessment Criterion Outside a Training Set

Abstract

The article addresses a commonly encountered problem of classification based on machine learning models. Given that attempts to classify objects outside the training sample are prone to yield unpredictable results, the classifiers may operate incorrectly on new data and may also be vulnerable to adversarial attacks. It is conjectured that these problems can be avoided provided that a sufficiently complete assessment of the classifier quality is made. The effectiveness of applying the conventional approach to estimating the classification quality is analyzed. Disadvantages of the conventional quality indicators, which do not allow one to evaluate the risk of errors and degree of machine learning model susceptibility to adversarial attacks, are described. А new classification quality criterion is proposed, which includes four characteristics: Excess, Deficit, Coating, and Approx (EDCA). The characteristics are quantified based on the ratio between the size of the space occupied by the training sample and the results of the classification of all points of the discretized space of features in the working range of their values. An experimental study for visual assessment and comparison of the quality of two multiclass SVM classifiers on characteristic synthetic data sets using the conventional and proposed quality indicators is carried out. The effectiveness and advantage of the newly introduced indicators in comparison with the conventional ones is demonstrated. Good interpretability of the quality indicator values, as well as the subjective consistency between the metrics and expected results from comparison of two SVM classifiers is confirmed. There is a reason to believe that application of the new approach to quality assessment will make it possible to construct more reliable classifiers based on machine learning.