Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality Simulator
Automated assessment of surgical skills is crucial for the successful training of junior surgeons. Twenty-three medical students followed a structured training curriculum on a laparoscopic virtual reality (VR) simulator. Three surgical tasks with significant educational merit were considered (Tasks...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-10-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/14/21/9677 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846173583714287616 |
|---|---|
| author | Konstantina Prevezanou Ioannis Seimenis Pantelis Karaiskos Emmanouil Pikoulis Panagis M. Lykoudis Constantinos Loukas |
| author_facet | Konstantina Prevezanou Ioannis Seimenis Pantelis Karaiskos Emmanouil Pikoulis Panagis M. Lykoudis Constantinos Loukas |
| author_sort | Konstantina Prevezanou |
| collection | DOAJ |
| description | Automated assessment of surgical skills is crucial for the successful training of junior surgeons. Twenty-three medical students followed a structured training curriculum on a laparoscopic virtual reality (VR) simulator. Three surgical tasks with significant educational merit were considered (Tasks 5, 6, and 7). We evaluated seven machine learning (ML) models for classifying the students’ trials into two and three classes based on the progress of training (Beginning vs. End and Beginning vs. Middle vs. End). Additionally, we evaluated the same ML framework and a deep learning approach (LSTM) for predicting the remaining number of trials required to complete the training proficiently. A model-agnostic technique from the domain of explainable artificial intelligence (XAI) was also utilized to obtain interpretations of the employed black-box ML classifiers. For 2-class classification, the best model showed an accuracy of 97.1%, 96.9%, and 75.7% for Task 5, 6, and 7, respectively, whereas for 3-class classification, the corresponding accuracy was 96.3%, 95.9%, and 99.7%, respectively. The best regression algorithm was LSTM with a Mean Absolute Error of 4 (Task 5) and 3.6 trials (Tasks 6, 7). According to XAI, the kinematic parameters have a stronger impact on the classification decision than the goal-oriented metrics. |
| format | Article |
| id | doaj-art-2e34a0d68b2e429ca0e9742c0b62f361 |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-2e34a0d68b2e429ca0e9742c0b62f3612024-11-08T14:32:59ZengMDPI AGApplied Sciences2076-34172024-10-011421967710.3390/app14219677Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality SimulatorKonstantina Prevezanou0Ioannis Seimenis1Pantelis Karaiskos2Emmanouil Pikoulis3Panagis M. Lykoudis4Constantinos Loukas5Laboratory of Medical Physics, Medical School, National and Kapodistrian University of Athens, 15772 Athens, GreeceLaboratory of Medical Physics, Medical School, National and Kapodistrian University of Athens, 15772 Athens, GreeceLaboratory of Medical Physics, Medical School, National and Kapodistrian University of Athens, 15772 Athens, Greece3rd Department of Surgery, University General Hospital “Attikon”, Medical School, National and Kapodistrian University of Athens, Rimini 1, 12461 Chaidari Attica, Greece3rd Department of Surgery, University General Hospital “Attikon”, Medical School, National and Kapodistrian University of Athens, Rimini 1, 12461 Chaidari Attica, GreeceLaboratory of Medical Physics, Medical School, National and Kapodistrian University of Athens, 15772 Athens, GreeceAutomated assessment of surgical skills is crucial for the successful training of junior surgeons. Twenty-three medical students followed a structured training curriculum on a laparoscopic virtual reality (VR) simulator. Three surgical tasks with significant educational merit were considered (Tasks 5, 6, and 7). We evaluated seven machine learning (ML) models for classifying the students’ trials into two and three classes based on the progress of training (Beginning vs. End and Beginning vs. Middle vs. End). Additionally, we evaluated the same ML framework and a deep learning approach (LSTM) for predicting the remaining number of trials required to complete the training proficiently. A model-agnostic technique from the domain of explainable artificial intelligence (XAI) was also utilized to obtain interpretations of the employed black-box ML classifiers. For 2-class classification, the best model showed an accuracy of 97.1%, 96.9%, and 75.7% for Task 5, 6, and 7, respectively, whereas for 3-class classification, the corresponding accuracy was 96.3%, 95.9%, and 99.7%, respectively. The best regression algorithm was LSTM with a Mean Absolute Error of 4 (Task 5) and 3.6 trials (Tasks 6, 7). According to XAI, the kinematic parameters have a stronger impact on the classification decision than the goal-oriented metrics.https://www.mdpi.com/2076-3417/14/21/9677minimally invasive surgerymachine learningVR simulatorskills assessment |
| spellingShingle | Konstantina Prevezanou Ioannis Seimenis Pantelis Karaiskos Emmanouil Pikoulis Panagis M. Lykoudis Constantinos Loukas Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality Simulator Applied Sciences minimally invasive surgery machine learning VR simulator skills assessment |
| title | Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality Simulator |
| title_full | Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality Simulator |
| title_fullStr | Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality Simulator |
| title_full_unstemmed | Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality Simulator |
| title_short | Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality Simulator |
| title_sort | machine learning approaches for evaluating the progress of surgical training on a virtual reality simulator |
| topic | minimally invasive surgery machine learning VR simulator skills assessment |
| url | https://www.mdpi.com/2076-3417/14/21/9677 |
| work_keys_str_mv | AT konstantinaprevezanou machinelearningapproachesforevaluatingtheprogressofsurgicaltrainingonavirtualrealitysimulator AT ioannisseimenis machinelearningapproachesforevaluatingtheprogressofsurgicaltrainingonavirtualrealitysimulator AT panteliskaraiskos machinelearningapproachesforevaluatingtheprogressofsurgicaltrainingonavirtualrealitysimulator AT emmanouilpikoulis machinelearningapproachesforevaluatingtheprogressofsurgicaltrainingonavirtualrealitysimulator AT panagismlykoudis machinelearningapproachesforevaluatingtheprogressofsurgicaltrainingonavirtualrealitysimulator AT constantinosloukas machinelearningapproachesforevaluatingtheprogressofsurgicaltrainingonavirtualrealitysimulator |