Improving Accuracy and Calibration of Deep Image Classifiers With Agreement-Driven Dynamic Ensemble
One of the biggest challenges when considering the applicability of Deep Learning systems to real-world problems is the possibility of failure in <italic>critical</italic> situations. Possible strategies to tackle this problem are two-fold: (i) models need to be highly accurate, conseque...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Open Journal of the Computer Society |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10806808/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841542520164581376 |
|---|---|
| author | Pedro Conde Rui L. Lopes Cristiano Premebida |
| author_facet | Pedro Conde Rui L. Lopes Cristiano Premebida |
| author_sort | Pedro Conde |
| collection | DOAJ |
| description | One of the biggest challenges when considering the applicability of Deep Learning systems to real-world problems is the possibility of failure in <italic>critical</italic> situations. Possible strategies to tackle this problem are two-fold: (i) models need to be highly accurate, consequently reducing this risk of failure; (ii) facing the impossibility of completely eliminating the risk of error, the models should be able to inform the level of uncertainty at the prediction level. As such, state-of-the-art DL models should be <italic>accurate</italic> and also <italic>calibrated</italic>, meaning that each prediction has to codify its confidence/uncertainty in a way that approximates the true likelihood of correctness. Nonetheless, relevant literature shows that improvements in <italic>accuracy</italic> and <italic>calibration</italic> are not usually related. This motivates the development of Agreement-Driven Dynamic Ensemble, a deep ensemble method that - by dynamically combining the advantages of two different ensemble strategies - is capable of achieving the highest possible accuracy values while obtaining also substantial improvements in calibration. The merits of the proposed algorithm are shown through a series of representative experiments, leveraging two different neural network architectures and three different datasets against multiple state-of-the-art baselines. |
| format | Article |
| id | doaj-art-09cb24e4b46a4f748b40dbf14c32adce |
| institution | Kabale University |
| issn | 2644-1268 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of the Computer Society |
| spelling | doaj-art-09cb24e4b46a4f748b40dbf14c32adce2025-01-14T00:02:54ZengIEEEIEEE Open Journal of the Computer Society2644-12682025-01-01616517610.1109/OJCS.2024.351998410806808Improving Accuracy and Calibration of Deep Image Classifiers With Agreement-Driven Dynamic EnsemblePedro Conde0https://orcid.org/0000-0003-0280-6705Rui L. Lopes1https://orcid.org/0000-0002-1579-1330Cristiano Premebida2https://orcid.org/0000-0002-2168-2077Institute of Systems and Robotics, Department of Electrical and Computer Engineering, University of Coimbra, Coimbra, PortugalCritical Software, Coimbra, S.A., PortugalInstitute of Systems and Robotics, Department of Electrical and Computer Engineering, University of Coimbra, Coimbra, PortugalOne of the biggest challenges when considering the applicability of Deep Learning systems to real-world problems is the possibility of failure in <italic>critical</italic> situations. Possible strategies to tackle this problem are two-fold: (i) models need to be highly accurate, consequently reducing this risk of failure; (ii) facing the impossibility of completely eliminating the risk of error, the models should be able to inform the level of uncertainty at the prediction level. As such, state-of-the-art DL models should be <italic>accurate</italic> and also <italic>calibrated</italic>, meaning that each prediction has to codify its confidence/uncertainty in a way that approximates the true likelihood of correctness. Nonetheless, relevant literature shows that improvements in <italic>accuracy</italic> and <italic>calibration</italic> are not usually related. This motivates the development of Agreement-Driven Dynamic Ensemble, a deep ensemble method that - by dynamically combining the advantages of two different ensemble strategies - is capable of achieving the highest possible accuracy values while obtaining also substantial improvements in calibration. The merits of the proposed algorithm are shown through a series of representative experiments, leveraging two different neural network architectures and three different datasets against multiple state-of-the-art baselines.https://ieeexplore.ieee.org/document/10806808/Deep ensemblesdeep learningimage classificationreliabilityprobabilistic interpretationuncertainty calibration |
| spellingShingle | Pedro Conde Rui L. Lopes Cristiano Premebida Improving Accuracy and Calibration of Deep Image Classifiers With Agreement-Driven Dynamic Ensemble IEEE Open Journal of the Computer Society Deep ensembles deep learning image classification reliability probabilistic interpretation uncertainty calibration |
| title | Improving Accuracy and Calibration of Deep Image Classifiers With Agreement-Driven Dynamic Ensemble |
| title_full | Improving Accuracy and Calibration of Deep Image Classifiers With Agreement-Driven Dynamic Ensemble |
| title_fullStr | Improving Accuracy and Calibration of Deep Image Classifiers With Agreement-Driven Dynamic Ensemble |
| title_full_unstemmed | Improving Accuracy and Calibration of Deep Image Classifiers With Agreement-Driven Dynamic Ensemble |
| title_short | Improving Accuracy and Calibration of Deep Image Classifiers With Agreement-Driven Dynamic Ensemble |
| title_sort | improving accuracy and calibration of deep image classifiers with agreement driven dynamic ensemble |
| topic | Deep ensembles deep learning image classification reliability probabilistic interpretation uncertainty calibration |
| url | https://ieeexplore.ieee.org/document/10806808/ |
| work_keys_str_mv | AT pedroconde improvingaccuracyandcalibrationofdeepimageclassifierswithagreementdrivendynamicensemble AT ruillopes improvingaccuracyandcalibrationofdeepimageclassifierswithagreementdrivendynamicensemble AT cristianopremebida improvingaccuracyandcalibrationofdeepimageclassifierswithagreementdrivendynamicensemble |