Why implementing machine learning algorithms in the clinic is not a plug-and-play solution: a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisResearch in context
Summary: Background: Artificial intelligence (AI) and machine learning (ML) algorithms have shown great promise in clinical medicine. Despite the increasing number of published algorithms, most remain unvalidated in real-world clinical settings. This study aims to simulate the practical implementat...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-01-01
|
| Series: | EBioMedicine |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2352396424005620 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846107647723438080 |
|---|---|
| author | Gernot Pucher Till Rostalski Felix Nensa Jens Kleesiek Hans Christian Reinhardt Christopher Martin Sauer |
| author_facet | Gernot Pucher Till Rostalski Felix Nensa Jens Kleesiek Hans Christian Reinhardt Christopher Martin Sauer |
| author_sort | Gernot Pucher |
| collection | DOAJ |
| description | Summary: Background: Artificial intelligence (AI) and machine learning (ML) algorithms have shown great promise in clinical medicine. Despite the increasing number of published algorithms, most remain unvalidated in real-world clinical settings. This study aims to simulate the practical implementation challenges of a recently developed ML algorithm, AI-PAL, designed for the diagnosis of acute leukaemia and report on its performance. Methods: We conducted a detailed simulation of the AI-PAL algorithm's implementation at the University Hospital Essen. Cohort building was performed using our Fast Healthcare Interoperability Resources (FHIR) database, identifying all initially diagnosed patients with acute leukaemia and selected differential diagnoses. The algorithm's performance was assessed by reproducing the original study's results. Findings: The AI-PAL algorithm demonstrated significantly lower performance in our simulated clinical implementation compared to prior published results. The area under the receiver operating characteristic curve for acute lymphoblastic leukaemia dropped to 0.67 (95% CI: 0.61–0.73) and for acute myeloid leukaemia to 0.71 (95% CI: 0.65–0.76). The recalibration of probability cutoffs determining confident diagnoses increased the number of confident positive diagnosis for acute leukaemia from 98 to 160, highlighting the necessity of local validation and adjustments. Interpretation: The findings underscore the challenges of implementing ML algorithms in clinical practice. Despite robust development and validation in research settings, ML models like AI-PAL may require significant adjustments and recalibration to maintain performance in different clinical settings. Our results suggest that clinical decision support algorithms should undergo local performance validation before integration into routine care to ensure reliability and safety. Funding: This study was supported by the DFG-cofounded UMEA Clinician Scientist Program and the Ministry of Culture and Science of the State of North Rhine-Westphalia. |
| format | Article |
| id | doaj-art-8faf00d83d7a45d3951b0b3b03ce2de0 |
| institution | Kabale University |
| issn | 2352-3964 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | EBioMedicine |
| spelling | doaj-art-8faf00d83d7a45d3951b0b3b03ce2de02024-12-26T08:56:36ZengElsevierEBioMedicine2352-39642025-01-01111105526Why implementing machine learning algorithms in the clinic is not a plug-and-play solution: a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisResearch in contextGernot Pucher0Till Rostalski1Felix Nensa2Jens Kleesiek3Hans Christian Reinhardt4Christopher Martin Sauer5Department of Haematology & Stem Cell Transplantation, West German Cancer Center, University Hospital Essen, Essen, Germany; Laboratory for Clinical Research and Real-World Evidence, Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, GermanyLaboratory for Clinical Research and Real-World Evidence, Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, GermanyInstitute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, GermanyInstitute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, GermanyDepartment of Haematology & Stem Cell Transplantation, West German Cancer Center, University Hospital Essen, Essen, GermanyDepartment of Haematology & Stem Cell Transplantation, West German Cancer Center, University Hospital Essen, Essen, Germany; Laboratory for Clinical Research and Real-World Evidence, Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany; Corresponding author. Department of Haematology & Stem Cell Transplantation, University Hospital Essen, Hufelandstr. 55, 45147, Essen, Germany.Summary: Background: Artificial intelligence (AI) and machine learning (ML) algorithms have shown great promise in clinical medicine. Despite the increasing number of published algorithms, most remain unvalidated in real-world clinical settings. This study aims to simulate the practical implementation challenges of a recently developed ML algorithm, AI-PAL, designed for the diagnosis of acute leukaemia and report on its performance. Methods: We conducted a detailed simulation of the AI-PAL algorithm's implementation at the University Hospital Essen. Cohort building was performed using our Fast Healthcare Interoperability Resources (FHIR) database, identifying all initially diagnosed patients with acute leukaemia and selected differential diagnoses. The algorithm's performance was assessed by reproducing the original study's results. Findings: The AI-PAL algorithm demonstrated significantly lower performance in our simulated clinical implementation compared to prior published results. The area under the receiver operating characteristic curve for acute lymphoblastic leukaemia dropped to 0.67 (95% CI: 0.61–0.73) and for acute myeloid leukaemia to 0.71 (95% CI: 0.65–0.76). The recalibration of probability cutoffs determining confident diagnoses increased the number of confident positive diagnosis for acute leukaemia from 98 to 160, highlighting the necessity of local validation and adjustments. Interpretation: The findings underscore the challenges of implementing ML algorithms in clinical practice. Despite robust development and validation in research settings, ML models like AI-PAL may require significant adjustments and recalibration to maintain performance in different clinical settings. Our results suggest that clinical decision support algorithms should undergo local performance validation before integration into routine care to ensure reliability and safety. Funding: This study was supported by the DFG-cofounded UMEA Clinician Scientist Program and the Ministry of Culture and Science of the State of North Rhine-Westphalia.http://www.sciencedirect.com/science/article/pii/S2352396424005620Machine learningArtificial intelligenceReal-world evaluationClinical implementationImplementation gap |
| spellingShingle | Gernot Pucher Till Rostalski Felix Nensa Jens Kleesiek Hans Christian Reinhardt Christopher Martin Sauer Why implementing machine learning algorithms in the clinic is not a plug-and-play solution: a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisResearch in context EBioMedicine Machine learning Artificial intelligence Real-world evaluation Clinical implementation Implementation gap |
| title | Why implementing machine learning algorithms in the clinic is not a plug-and-play solution: a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisResearch in context |
| title_full | Why implementing machine learning algorithms in the clinic is not a plug-and-play solution: a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisResearch in context |
| title_fullStr | Why implementing machine learning algorithms in the clinic is not a plug-and-play solution: a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisResearch in context |
| title_full_unstemmed | Why implementing machine learning algorithms in the clinic is not a plug-and-play solution: a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisResearch in context |
| title_short | Why implementing machine learning algorithms in the clinic is not a plug-and-play solution: a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisResearch in context |
| title_sort | why implementing machine learning algorithms in the clinic is not a plug and play solution a simulation study of a machine learning algorithm for acute leukaemia subtype diagnosisresearch in context |
| topic | Machine learning Artificial intelligence Real-world evaluation Clinical implementation Implementation gap |
| url | http://www.sciencedirect.com/science/article/pii/S2352396424005620 |
| work_keys_str_mv | AT gernotpucher whyimplementingmachinelearningalgorithmsintheclinicisnotaplugandplaysolutionasimulationstudyofamachinelearningalgorithmforacuteleukaemiasubtypediagnosisresearchincontext AT tillrostalski whyimplementingmachinelearningalgorithmsintheclinicisnotaplugandplaysolutionasimulationstudyofamachinelearningalgorithmforacuteleukaemiasubtypediagnosisresearchincontext AT felixnensa whyimplementingmachinelearningalgorithmsintheclinicisnotaplugandplaysolutionasimulationstudyofamachinelearningalgorithmforacuteleukaemiasubtypediagnosisresearchincontext AT jenskleesiek whyimplementingmachinelearningalgorithmsintheclinicisnotaplugandplaysolutionasimulationstudyofamachinelearningalgorithmforacuteleukaemiasubtypediagnosisresearchincontext AT hanschristianreinhardt whyimplementingmachinelearningalgorithmsintheclinicisnotaplugandplaysolutionasimulationstudyofamachinelearningalgorithmforacuteleukaemiasubtypediagnosisresearchincontext AT christophermartinsauer whyimplementingmachinelearningalgorithmsintheclinicisnotaplugandplaysolutionasimulationstudyofamachinelearningalgorithmforacuteleukaemiasubtypediagnosisresearchincontext |