Continuous Speech-Based Fatigue Detection and Transition State Prediction for Air Traffic Controllers
Air traffic controllers (ATC) play a critical role in ensuring aviation safety, but their demanding workload can lead to fatigue, potentially compromising their performance. This paper presents a study that investigates speech features responsible for detecting ATC fatigue and proposes an approach t...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10818676/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841554026735337472 |
|---|---|
| author | Susmitha Vekkot Surya Teja Chavali Charan Tej Kandavalli Rama Sai Abhishek Podila Deepa Gupta Mohammed Zakariah Yousef Ajami Alotaibi |
| author_facet | Susmitha Vekkot Surya Teja Chavali Charan Tej Kandavalli Rama Sai Abhishek Podila Deepa Gupta Mohammed Zakariah Yousef Ajami Alotaibi |
| author_sort | Susmitha Vekkot |
| collection | DOAJ |
| description | Air traffic controllers (ATC) play a critical role in ensuring aviation safety, but their demanding workload can lead to fatigue, potentially compromising their performance. This paper presents a study that investigates speech features responsible for detecting ATC fatigue and proposes an approach to predict the timestamp at which an ATC transitions into a fatigue state from a continuous speech sample. The main contributions of this work are the creation of a continuous speech ATC dataset and the identification of a lightweight optimum feature set for fatigue classification from ATC speech. For the initial task, the classification of raw speech signals into fatigue and non-fatigue categories was performed using the top-10 best features selected from the openSMILE feature set. The evaluation was carried out using various learning algorithms such as XGBoost, Adaboost, Random Forest, HistogramGB, and 1D-CNN. The ensemble algorithms demonstrated the best performance, achieving a maximum accuracy of 100% on the XGBoost test set. Further, interpretability was analyzed using the SHAP tool, which identified the prominent features for the task. The second task involved creating a continuous speech dataset comprising approximately 18,900 samples from the ATC corpus, with an average duration of 63-65 seconds per sample. The continuous speech samples were prepared by the randomized concatenation of fatigue and non-fatigue chunks, each with a duration of approximately 15 seconds. Automated sequence labeling was performed on uniformly segmented continuous speech samples. MFCC and statistical features were extracted from the labelled continuous speech and input into various Recurrent Neural Networks, such as bi-LSTM and Bi-GRU, for fatigue state prediction tasks. A combination of these features using bi-LSTM modeling achieved a maximum precision, recall, F-score, and average accuracy of 99% each. Finally, sample-wise timestamp prediction was performed using the labels: fatigue, non-fatigue, and ambiguous (transition). To the best of the authors’ knowledge, this research is the first of its kind to address continuous speech-based fatigue state prediction for ATCs. All tasks were conducted using the Civil Aviation Administration of China ATC corpus. |
| format | Article |
| id | doaj-art-3dcad620fc634f4888ec11624267a40b |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-3dcad620fc634f4888ec11624267a40b2025-01-09T00:01:25ZengIEEEIEEE Access2169-35362025-01-01133298331910.1109/ACCESS.2024.352445210818676Continuous Speech-Based Fatigue Detection and Transition State Prediction for Air Traffic ControllersSusmitha Vekkot0https://orcid.org/0000-0002-6638-214XSurya Teja Chavali1https://orcid.org/0000-0003-3214-7245Charan Tej Kandavalli2Rama Sai Abhishek Podila3Deepa Gupta4https://orcid.org/0000-0002-1041-5125Mohammed Zakariah5https://orcid.org/0000-0002-2488-2605Yousef Ajami Alotaibi6https://orcid.org/0000-0003-0998-8978Department of Electronics and Communication Engineering, Amrita School of Engineering Bengaluru, Amrita Vishwa Vidyapeetham, Bengaluru, IndiaDepartment of Computer Science and Engineering, Amrita School of Computing Bengaluru, Amrita Vishwa Vidyapeetham, Bengaluru, IndiaDepartment of Computer Science and Engineering, Amrita School of Computing Bengaluru, Amrita Vishwa Vidyapeetham, Bengaluru, IndiaDepartment of Computer Science and Engineering, Amrita School of Computing Bengaluru, Amrita Vishwa Vidyapeetham, Bengaluru, IndiaDepartment of Computer Science and Engineering, Amrita School of Computing Bengaluru, Amrita Vishwa Vidyapeetham, Bengaluru, IndiaDepartment of Computer Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaDepartment of Computer Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaAir traffic controllers (ATC) play a critical role in ensuring aviation safety, but their demanding workload can lead to fatigue, potentially compromising their performance. This paper presents a study that investigates speech features responsible for detecting ATC fatigue and proposes an approach to predict the timestamp at which an ATC transitions into a fatigue state from a continuous speech sample. The main contributions of this work are the creation of a continuous speech ATC dataset and the identification of a lightweight optimum feature set for fatigue classification from ATC speech. For the initial task, the classification of raw speech signals into fatigue and non-fatigue categories was performed using the top-10 best features selected from the openSMILE feature set. The evaluation was carried out using various learning algorithms such as XGBoost, Adaboost, Random Forest, HistogramGB, and 1D-CNN. The ensemble algorithms demonstrated the best performance, achieving a maximum accuracy of 100% on the XGBoost test set. Further, interpretability was analyzed using the SHAP tool, which identified the prominent features for the task. The second task involved creating a continuous speech dataset comprising approximately 18,900 samples from the ATC corpus, with an average duration of 63-65 seconds per sample. The continuous speech samples were prepared by the randomized concatenation of fatigue and non-fatigue chunks, each with a duration of approximately 15 seconds. Automated sequence labeling was performed on uniformly segmented continuous speech samples. MFCC and statistical features were extracted from the labelled continuous speech and input into various Recurrent Neural Networks, such as bi-LSTM and Bi-GRU, for fatigue state prediction tasks. A combination of these features using bi-LSTM modeling achieved a maximum precision, recall, F-score, and average accuracy of 99% each. Finally, sample-wise timestamp prediction was performed using the labels: fatigue, non-fatigue, and ambiguous (transition). To the best of the authors’ knowledge, this research is the first of its kind to address continuous speech-based fatigue state prediction for ATCs. All tasks were conducted using the Civil Aviation Administration of China ATC corpus.https://ieeexplore.ieee.org/document/10818676/Air traffic controllercontinuous speechXGBoostrecurrent neural networksbi-LSTMMFCC |
| spellingShingle | Susmitha Vekkot Surya Teja Chavali Charan Tej Kandavalli Rama Sai Abhishek Podila Deepa Gupta Mohammed Zakariah Yousef Ajami Alotaibi Continuous Speech-Based Fatigue Detection and Transition State Prediction for Air Traffic Controllers IEEE Access Air traffic controller continuous speech XGBoost recurrent neural networks bi-LSTM MFCC |
| title | Continuous Speech-Based Fatigue Detection and Transition State Prediction for Air Traffic Controllers |
| title_full | Continuous Speech-Based Fatigue Detection and Transition State Prediction for Air Traffic Controllers |
| title_fullStr | Continuous Speech-Based Fatigue Detection and Transition State Prediction for Air Traffic Controllers |
| title_full_unstemmed | Continuous Speech-Based Fatigue Detection and Transition State Prediction for Air Traffic Controllers |
| title_short | Continuous Speech-Based Fatigue Detection and Transition State Prediction for Air Traffic Controllers |
| title_sort | continuous speech based fatigue detection and transition state prediction for air traffic controllers |
| topic | Air traffic controller continuous speech XGBoost recurrent neural networks bi-LSTM MFCC |
| url | https://ieeexplore.ieee.org/document/10818676/ |
| work_keys_str_mv | AT susmithavekkot continuousspeechbasedfatiguedetectionandtransitionstatepredictionforairtrafficcontrollers AT suryatejachavali continuousspeechbasedfatiguedetectionandtransitionstatepredictionforairtrafficcontrollers AT charantejkandavalli continuousspeechbasedfatiguedetectionandtransitionstatepredictionforairtrafficcontrollers AT ramasaiabhishekpodila continuousspeechbasedfatiguedetectionandtransitionstatepredictionforairtrafficcontrollers AT deepagupta continuousspeechbasedfatiguedetectionandtransitionstatepredictionforairtrafficcontrollers AT mohammedzakariah continuousspeechbasedfatiguedetectionandtransitionstatepredictionforairtrafficcontrollers AT yousefajamialotaibi continuousspeechbasedfatiguedetectionandtransitionstatepredictionforairtrafficcontrollers |