A non-anatomical graph structure for boundary detection in continuous sign language
Abstract Recently, the challenge of the boundary detection of isolated signs in a continuous sign video has been studied by researchers. To enhance the model performance, replace the handcrafted feature extractor, and also consider the hand structure in these models, we propose a deep learning-based...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-07-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-11598-3 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849332826804060160 |
|---|---|
| author | Razieh Rastgoo Kourosh Kiani Sergio Escalera |
| author_facet | Razieh Rastgoo Kourosh Kiani Sergio Escalera |
| author_sort | Razieh Rastgoo |
| collection | DOAJ |
| description | Abstract Recently, the challenge of the boundary detection of isolated signs in a continuous sign video has been studied by researchers. To enhance the model performance, replace the handcrafted feature extractor, and also consider the hand structure in these models, we propose a deep learning-based approach, including a combination of the Graph Convolutional Network (GCN) and the Transformer models, along with a post-processing mechanism for final boundary detection. More specifically, the proposed approach includes two main steps: Pre-training on the isolated sign videos and Deploying on the continuous sign videos. In the first step, the enriched spatial features obtained from the GCN model are fed to the Transformer model to push the temporal information in the video stream. This model in pre-trained only using the pre-processed isolated sign videos with same frame lengths. During the second step, the sliding window method with the pre-defined window size is moved on the continuous sign video, including the un-processed isolated sign videos with different frame lengths. More concretely, the content of each window is processed using the pre-trained model obtained from the first step and the class probabilities of the Fully Connected (FC) layer embedded in the Transformer model are fed to the post-processing module, which aims to detect the accurate boundary of the un-processed isolated signs. In addition, we propose to present a non-anatomical graph structure to better present the hand joints movements and relations during the signing. Relying on the proposed non-anatomical hand graph structure as well as the self-attention mechanism in the Transformer model, the proposed model can successfully tackle the challenges of boundary detection in continuous sign videos. Experimental results on two datasets show the superiority of the proposed model in dealing with isolated sign boundary detection in continuous sign sequences. |
| format | Article |
| id | doaj-art-a2a4d49a124e4dab99ec28eef2b60ff1 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-a2a4d49a124e4dab99ec28eef2b60ff12025-08-20T03:46:05ZengNature PortfolioScientific Reports2045-23222025-07-0115111410.1038/s41598-025-11598-3A non-anatomical graph structure for boundary detection in continuous sign languageRazieh Rastgoo0Kourosh Kiani1Sergio Escalera2Electrical and Computer Engineering Department, Semnan UniversityElectrical and Computer Engineering Department, Semnan UniversityDepartment of Mathematics and Informatics, University of Barcelona and Computer Vision CenterAbstract Recently, the challenge of the boundary detection of isolated signs in a continuous sign video has been studied by researchers. To enhance the model performance, replace the handcrafted feature extractor, and also consider the hand structure in these models, we propose a deep learning-based approach, including a combination of the Graph Convolutional Network (GCN) and the Transformer models, along with a post-processing mechanism for final boundary detection. More specifically, the proposed approach includes two main steps: Pre-training on the isolated sign videos and Deploying on the continuous sign videos. In the first step, the enriched spatial features obtained from the GCN model are fed to the Transformer model to push the temporal information in the video stream. This model in pre-trained only using the pre-processed isolated sign videos with same frame lengths. During the second step, the sliding window method with the pre-defined window size is moved on the continuous sign video, including the un-processed isolated sign videos with different frame lengths. More concretely, the content of each window is processed using the pre-trained model obtained from the first step and the class probabilities of the Fully Connected (FC) layer embedded in the Transformer model are fed to the post-processing module, which aims to detect the accurate boundary of the un-processed isolated signs. In addition, we propose to present a non-anatomical graph structure to better present the hand joints movements and relations during the signing. Relying on the proposed non-anatomical hand graph structure as well as the self-attention mechanism in the Transformer model, the proposed model can successfully tackle the challenges of boundary detection in continuous sign videos. Experimental results on two datasets show the superiority of the proposed model in dealing with isolated sign boundary detection in continuous sign sequences.https://doi.org/10.1038/s41598-025-11598-3Continuous sign sequenceIsolated sign recognitionGraph convolutional network (GCN)TransformerBoundary detection |
| spellingShingle | Razieh Rastgoo Kourosh Kiani Sergio Escalera A non-anatomical graph structure for boundary detection in continuous sign language Scientific Reports Continuous sign sequence Isolated sign recognition Graph convolutional network (GCN) Transformer Boundary detection |
| title | A non-anatomical graph structure for boundary detection in continuous sign language |
| title_full | A non-anatomical graph structure for boundary detection in continuous sign language |
| title_fullStr | A non-anatomical graph structure for boundary detection in continuous sign language |
| title_full_unstemmed | A non-anatomical graph structure for boundary detection in continuous sign language |
| title_short | A non-anatomical graph structure for boundary detection in continuous sign language |
| title_sort | non anatomical graph structure for boundary detection in continuous sign language |
| topic | Continuous sign sequence Isolated sign recognition Graph convolutional network (GCN) Transformer Boundary detection |
| url | https://doi.org/10.1038/s41598-025-11598-3 |
| work_keys_str_mv | AT raziehrastgoo anonanatomicalgraphstructureforboundarydetectionincontinuoussignlanguage AT kouroshkiani anonanatomicalgraphstructureforboundarydetectionincontinuoussignlanguage AT sergioescalera anonanatomicalgraphstructureforboundarydetectionincontinuoussignlanguage AT raziehrastgoo nonanatomicalgraphstructureforboundarydetectionincontinuoussignlanguage AT kouroshkiani nonanatomicalgraphstructureforboundarydetectionincontinuoussignlanguage AT sergioescalera nonanatomicalgraphstructureforboundarydetectionincontinuoussignlanguage |