RETRACTED ARTICLE: Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network
Abstract Sign Language Recognition is a breakthrough for communication among deaf-mute society and has been a critical research topic for years. Although some of the previous studies have successfully recognized sign language, it requires many costly instruments including sensors, devices, and high-...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2023-10-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-023-43852-x |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841544612636786688 |
|---|---|
| author | Refat Khan Pathan Munmun Biswas Suraiya Yasmin Mayeen Uddin Khandaker Mohammad Salman Ahmed A. F. Youssef |
| author_facet | Refat Khan Pathan Munmun Biswas Suraiya Yasmin Mayeen Uddin Khandaker Mohammad Salman Ahmed A. F. Youssef |
| author_sort | Refat Khan Pathan |
| collection | DOAJ |
| description | Abstract Sign Language Recognition is a breakthrough for communication among deaf-mute society and has been a critical research topic for years. Although some of the previous studies have successfully recognized sign language, it requires many costly instruments including sensors, devices, and high-end processing power. However, such drawbacks can be easily overcome by employing artificial intelligence-based techniques. Since, in this modern era of advanced mobile technology, using a camera to take video or images is much easier, this study demonstrates a cost-effective technique to detect American Sign Language (ASL) using an image dataset. Here, “Finger Spelling, A” dataset has been used, with 24 letters (except j and z as they contain motion). The main reason for using this dataset is that these images have a complex background with different environments and scene colors. Two layers of image processing have been used: in the first layer, images are processed as a whole for training, and in the second layer, the hand landmarks are extracted. A multi-headed convolutional neural network (CNN) model has been proposed and tested with 30% of the dataset to train these two layers. To avoid the overfitting problem, data augmentation and dynamic learning rate reduction have been used. With the proposed model, 98.981% test accuracy has been achieved. It is expected that this study may help to develop an efficient human–machine communication system for a deaf-mute society. |
| format | Article |
| id | doaj-art-ae0bfc8b88a24c3badd8ccb06a61ee6c |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2023-10-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-ae0bfc8b88a24c3badd8ccb06a61ee6c2025-01-12T12:25:24ZengNature PortfolioScientific Reports2045-23222023-10-0113111110.1038/s41598-023-43852-xRETRACTED ARTICLE: Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural networkRefat Khan Pathan0Munmun Biswas1Suraiya Yasmin2Mayeen Uddin Khandaker3Mohammad Salman4Ahmed A. F. Youssef5Department of Computing and Information Systems, School of Engineering and Technology, Sunway UniversityDepartment of Computer Science and Engineering, BGC Trust University BangladeshDepartment of Computer and Information Science, Graduate School of Engineering, Tokyo University of Agriculture and TechnologyCentre for Applied Physics and Radiation Technologies, School of Engineering and Technology, Sunway UniversityCollege of Engineering and Technology, American University of the Middle EastCollege of Engineering and Technology, American University of the Middle EastAbstract Sign Language Recognition is a breakthrough for communication among deaf-mute society and has been a critical research topic for years. Although some of the previous studies have successfully recognized sign language, it requires many costly instruments including sensors, devices, and high-end processing power. However, such drawbacks can be easily overcome by employing artificial intelligence-based techniques. Since, in this modern era of advanced mobile technology, using a camera to take video or images is much easier, this study demonstrates a cost-effective technique to detect American Sign Language (ASL) using an image dataset. Here, “Finger Spelling, A” dataset has been used, with 24 letters (except j and z as they contain motion). The main reason for using this dataset is that these images have a complex background with different environments and scene colors. Two layers of image processing have been used: in the first layer, images are processed as a whole for training, and in the second layer, the hand landmarks are extracted. A multi-headed convolutional neural network (CNN) model has been proposed and tested with 30% of the dataset to train these two layers. To avoid the overfitting problem, data augmentation and dynamic learning rate reduction have been used. With the proposed model, 98.981% test accuracy has been achieved. It is expected that this study may help to develop an efficient human–machine communication system for a deaf-mute society.https://doi.org/10.1038/s41598-023-43852-x |
| spellingShingle | Refat Khan Pathan Munmun Biswas Suraiya Yasmin Mayeen Uddin Khandaker Mohammad Salman Ahmed A. F. Youssef RETRACTED ARTICLE: Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network Scientific Reports |
| title | RETRACTED ARTICLE: Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network |
| title_full | RETRACTED ARTICLE: Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network |
| title_fullStr | RETRACTED ARTICLE: Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network |
| title_full_unstemmed | RETRACTED ARTICLE: Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network |
| title_short | RETRACTED ARTICLE: Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network |
| title_sort | retracted article sign language recognition using the fusion of image and hand landmarks through multi headed convolutional neural network |
| url | https://doi.org/10.1038/s41598-023-43852-x |
| work_keys_str_mv | AT refatkhanpathan retractedarticlesignlanguagerecognitionusingthefusionofimageandhandlandmarksthroughmultiheadedconvolutionalneuralnetwork AT munmunbiswas retractedarticlesignlanguagerecognitionusingthefusionofimageandhandlandmarksthroughmultiheadedconvolutionalneuralnetwork AT suraiyayasmin retractedarticlesignlanguagerecognitionusingthefusionofimageandhandlandmarksthroughmultiheadedconvolutionalneuralnetwork AT mayeenuddinkhandaker retractedarticlesignlanguagerecognitionusingthefusionofimageandhandlandmarksthroughmultiheadedconvolutionalneuralnetwork AT mohammadsalman retractedarticlesignlanguagerecognitionusingthefusionofimageandhandlandmarksthroughmultiheadedconvolutionalneuralnetwork AT ahmedafyoussef retractedarticlesignlanguagerecognitionusingthefusionofimageandhandlandmarksthroughmultiheadedconvolutionalneuralnetwork |