Comparative study of quality estimation of binary classification
The paper describes results of analytical and experimental analysis of seventeen functions used for evaluation of binary classification results of arbitrary data. The results are presented by 2×2 error matrices. The behavior and properties of the main functions calculated by the elements of such mat...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | Russian |
| Published: |
National Academy of Sciences of Belarus, the United Institute of Informatics Problems
2020-03-01
|
| Series: | Informatika |
| Subjects: | |
| Online Access: | https://inf.grid.by/jour/article/view/1044 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849240220504948736 |
|---|---|
| author | V. V. Starovoitov Yu. I. Golub |
| author_facet | V. V. Starovoitov Yu. I. Golub |
| author_sort | V. V. Starovoitov |
| collection | DOAJ |
| description | The paper describes results of analytical and experimental analysis of seventeen functions used for evaluation of binary classification results of arbitrary data. The results are presented by 2×2 error matrices. The behavior and properties of the main functions calculated by the elements of such matrices are studied. Classification options with balanced and imbalanced datasets are analyzed. It is shown that there are linear dependencies between some functions, many functions are invariant to the transposition of the error matrix, which allows us to calculate the estimation without specifying the order in which their elements were written to the matrices.It has been proven that all classical measures such as Sensitivity, Specificity, Precision, Accuracy, F1, F2, GM, the Jacquard index are sensitive to the imbalance of classified data and distort estimation of smaller class objects classification errors. Sensitivity to imbalance is found in the Matthews correlation coefficient and Kohen’s kappa. It has been experimentally shown that functions such as the confusion entropy, the discriminatory power, and the diagnostic odds ratio should not be used for analysis of binary classification of imbalanced datasets. The last two functions are invariant to the imbalance of classified data, but poorly evaluate results with approximately equal common percentage of classification errors in two classes.We proved that the area under the ROC curve (AUC) and the Yuden index calculated from the binary classification confusion matrix are linearly dependent and are the best estimation functions of both balanced and imbalanced datasets. |
| format | Article |
| id | doaj-art-72524e5cf24f46d1a618663b8f56f96f |
| institution | Kabale University |
| issn | 1816-0301 |
| language | Russian |
| publishDate | 2020-03-01 |
| publisher | National Academy of Sciences of Belarus, the United Institute of Informatics Problems |
| record_format | Article |
| series | Informatika |
| spelling | doaj-art-72524e5cf24f46d1a618663b8f56f96f2025-08-20T04:00:40ZrusNational Academy of Sciences of Belarus, the United Institute of Informatics ProblemsInformatika1816-03012020-03-011718710110.37661/1816-0301-2020-17-1-87-101914Comparative study of quality estimation of binary classificationV. V. Starovoitov0Yu. I. Golub1The United Institute of Informatics Problems of the National Academy of Sciences of BelarusThe United Institute of Informatics Problems of the National Academy of Sciences of BelarusThe paper describes results of analytical and experimental analysis of seventeen functions used for evaluation of binary classification results of arbitrary data. The results are presented by 2×2 error matrices. The behavior and properties of the main functions calculated by the elements of such matrices are studied. Classification options with balanced and imbalanced datasets are analyzed. It is shown that there are linear dependencies between some functions, many functions are invariant to the transposition of the error matrix, which allows us to calculate the estimation without specifying the order in which their elements were written to the matrices.It has been proven that all classical measures such as Sensitivity, Specificity, Precision, Accuracy, F1, F2, GM, the Jacquard index are sensitive to the imbalance of classified data and distort estimation of smaller class objects classification errors. Sensitivity to imbalance is found in the Matthews correlation coefficient and Kohen’s kappa. It has been experimentally shown that functions such as the confusion entropy, the discriminatory power, and the diagnostic odds ratio should not be used for analysis of binary classification of imbalanced datasets. The last two functions are invariant to the imbalance of classified data, but poorly evaluate results with approximately equal common percentage of classification errors in two classes.We proved that the area under the ROC curve (AUC) and the Yuden index calculated from the binary classification confusion matrix are linearly dependent and are the best estimation functions of both balanced and imbalanced datasets.https://inf.grid.by/jour/article/view/1044binary classificationconfusion matrixfunctions of accuracy classificationarea under roc curveyouden’s index |
| spellingShingle | V. V. Starovoitov Yu. I. Golub Comparative study of quality estimation of binary classification Informatika binary classification confusion matrix functions of accuracy classification area under roc curve youden’s index |
| title | Comparative study of quality estimation of binary classification |
| title_full | Comparative study of quality estimation of binary classification |
| title_fullStr | Comparative study of quality estimation of binary classification |
| title_full_unstemmed | Comparative study of quality estimation of binary classification |
| title_short | Comparative study of quality estimation of binary classification |
| title_sort | comparative study of quality estimation of binary classification |
| topic | binary classification confusion matrix functions of accuracy classification area under roc curve youden’s index |
| url | https://inf.grid.by/jour/article/view/1044 |
| work_keys_str_mv | AT vvstarovoitov comparativestudyofqualityestimationofbinaryclassification AT yuigolub comparativestudyofqualityestimationofbinaryclassification |