Sign Language Recognition Based on CNN-BiLSTM Using RF Signals
As the number of individuals with speech and hearing impairments continues to grow, the demand for sign language recognition systems is increasing. Furthermore, most radio frequency-based gesture recognition systems have primarily focused on simple gestures, neglecting more complex actions such as s...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10798415/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841533407473958912 |
|---|---|
| author | Yajun Zhang Yuankang Wang Feng Li Weiqian Yu Congcong Wang Ying Jiang |
| author_facet | Yajun Zhang Yuankang Wang Feng Li Weiqian Yu Congcong Wang Ying Jiang |
| author_sort | Yajun Zhang |
| collection | DOAJ |
| description | As the number of individuals with speech and hearing impairments continues to grow, the demand for sign language recognition systems is increasing. Furthermore, most radio frequency-based gesture recognition systems have primarily focused on simple gestures, neglecting more complex actions such as sign language. Therefore, this paper proposes RF-SL, a commercial RFID-based contactless sign language recognition system. This system does not require users to wear any tags and only requires them to perform sign language gestures positioned between an antenna and a fixed multi-tag array to collect sign language signals. Firstly, by removing static reflection noise, we reduce the impact of environmental noise and obtain purer data. Secondly, to accurately delineate the start and end points of sign language action signals, we propose an improved version of the Varri signal segmentation algorithm called Varri+ to effectively optimize the segmentation of sign language actions. Thirdly, we combine the CNN model, known for its robust feature extraction capabilities, with the BiLSTM model, which excels in feature fusion. Extensive testing and evaluation were conducted in two realistic scenarios. The experimental results show that RF-SL achieves an overall average recognition accuracy of 96.8%, with an average recognition accuracy of 96.3% for new users. Additionally, RF-SL achieves an F1 score of 97.5% in an empty room with the weak multipath effects and an F1 score of 93.6% in a classroom with the strong multipath effects. These results demonstrate that our system is dynamic, flexible and robust in sign language recognition. |
| format | Article |
| id | doaj-art-550b4d0a97a6432f948e4b4dc525bd8a |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-550b4d0a97a6432f948e4b4dc525bd8a2025-01-16T00:01:28ZengIEEEIEEE Access2169-35362024-01-011219048719050410.1109/ACCESS.2024.351741710798415Sign Language Recognition Based on CNN-BiLSTM Using RF SignalsYajun Zhang0https://orcid.org/0000-0001-9396-3313Yuankang Wang1https://orcid.org/0009-0006-6303-9414Feng Li2Weiqian Yu3https://orcid.org/0009-0002-4805-9637Congcong Wang4https://orcid.org/0009-0003-3673-4065Ying Jiang5https://orcid.org/0000-0003-0300-6101School of Software, Xinjiang University, Ürümqi, ChinaSchool of Software, Xinjiang University, Ürümqi, ChinaXinjiang Uygur Autonomous Research Institute of Measurement and Testing, Ürümqi, ChinaSchool of Software, Xinjiang University, Ürümqi, ChinaSchool of Software, Xinjiang University, Ürümqi, ChinaSchool of Software, Xinjiang University, Ürümqi, ChinaAs the number of individuals with speech and hearing impairments continues to grow, the demand for sign language recognition systems is increasing. Furthermore, most radio frequency-based gesture recognition systems have primarily focused on simple gestures, neglecting more complex actions such as sign language. Therefore, this paper proposes RF-SL, a commercial RFID-based contactless sign language recognition system. This system does not require users to wear any tags and only requires them to perform sign language gestures positioned between an antenna and a fixed multi-tag array to collect sign language signals. Firstly, by removing static reflection noise, we reduce the impact of environmental noise and obtain purer data. Secondly, to accurately delineate the start and end points of sign language action signals, we propose an improved version of the Varri signal segmentation algorithm called Varri+ to effectively optimize the segmentation of sign language actions. Thirdly, we combine the CNN model, known for its robust feature extraction capabilities, with the BiLSTM model, which excels in feature fusion. Extensive testing and evaluation were conducted in two realistic scenarios. The experimental results show that RF-SL achieves an overall average recognition accuracy of 96.8%, with an average recognition accuracy of 96.3% for new users. Additionally, RF-SL achieves an F1 score of 97.5% in an empty room with the weak multipath effects and an F1 score of 93.6% in a classroom with the strong multipath effects. These results demonstrate that our system is dynamic, flexible and robust in sign language recognition.https://ieeexplore.ieee.org/document/10798415/Sign language recognitionRF signalssignal segmentationRFIDCNNBiLSTM |
| spellingShingle | Yajun Zhang Yuankang Wang Feng Li Weiqian Yu Congcong Wang Ying Jiang Sign Language Recognition Based on CNN-BiLSTM Using RF Signals IEEE Access Sign language recognition RF signals signal segmentation RFID CNN BiLSTM |
| title | Sign Language Recognition Based on CNN-BiLSTM Using RF Signals |
| title_full | Sign Language Recognition Based on CNN-BiLSTM Using RF Signals |
| title_fullStr | Sign Language Recognition Based on CNN-BiLSTM Using RF Signals |
| title_full_unstemmed | Sign Language Recognition Based on CNN-BiLSTM Using RF Signals |
| title_short | Sign Language Recognition Based on CNN-BiLSTM Using RF Signals |
| title_sort | sign language recognition based on cnn bilstm using rf signals |
| topic | Sign language recognition RF signals signal segmentation RFID CNN BiLSTM |
| url | https://ieeexplore.ieee.org/document/10798415/ |
| work_keys_str_mv | AT yajunzhang signlanguagerecognitionbasedoncnnbilstmusingrfsignals AT yuankangwang signlanguagerecognitionbasedoncnnbilstmusingrfsignals AT fengli signlanguagerecognitionbasedoncnnbilstmusingrfsignals AT weiqianyu signlanguagerecognitionbasedoncnnbilstmusingrfsignals AT congcongwang signlanguagerecognitionbasedoncnnbilstmusingrfsignals AT yingjiang signlanguagerecognitionbasedoncnnbilstmusingrfsignals |