TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor [version 1; peer review: 2 approved]
Background: Intelligent monitoring of human action in production is an important step to help standardize production processes and construct a digital twin shop-floor rapidly. Human action has a significant impact on the production safety and efficiency of a shop-floor, however, because of the high...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
F1000 Research Ltd
2021-12-01
|
| Series: | Digital Twin |
| Subjects: | |
| Online Access: | https://digitaltwin1.org/articles/1-10/v1 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846126263087923200 |
|---|---|
| author | Liang Fu Yunfeng Xie Tingyu Liu Yifeng Sun Qing Hong |
| author_facet | Liang Fu Yunfeng Xie Tingyu Liu Yifeng Sun Qing Hong |
| author_sort | Liang Fu |
| collection | DOAJ |
| description | Background: Intelligent monitoring of human action in production is an important step to help standardize production processes and construct a digital twin shop-floor rapidly. Human action has a significant impact on the production safety and efficiency of a shop-floor, however, because of the high individual initiative of humans, it is difficult to realize real-time action detection in a digital twin shop-floor. Methods: We proposed a real-time detection approach for shop-floor production action. This approach used the sequence data of continuous human skeleton joints sequences as the input. We then reconstructed the Joint Classification-Regression Recurrent Neural Networks (JCR-RNN) based on Temporal Convolution Network (TCN) and Graph Convolution Network (GCN). We called this approach the Temporal Action Detection Net (TAD-Net), which realized real-time shop-floor production action detection. Results: The results of the verification experiment showed that our approach has achieved a high temporal positioning score, recognition speed, and accuracy when applied to the existing Online Action Detection (OAD) dataset and the Nanjing University of Science and Technology 3 Dimensions (NJUST3D) dataset. TAD-Net can meet the actual needs of the digital twin shop-floor. Conclusions: Our method has higher recognition accuracy, temporal positioning accuracy, and faster running speed than other mainstream network models, it can better meet actual application requirements, and has important research value and practical significance for standardizing shop-floor production processes, reducing production security risks, and contributing to the understanding of real-time production action. |
| format | Article |
| id | doaj-art-225e4c35cea44c67b320f85a12c08f93 |
| institution | Kabale University |
| issn | 2752-5783 |
| language | English |
| publishDate | 2021-12-01 |
| publisher | F1000 Research Ltd |
| record_format | Article |
| series | Digital Twin |
| spelling | doaj-art-225e4c35cea44c67b320f85a12c08f932024-12-13T01:00:04ZengF1000 Research LtdDigital Twin2752-57832021-12-01118683TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor [version 1; peer review: 2 approved]Liang Fu0Yunfeng Xie1Tingyu Liu2https://orcid.org/0000-0001-8934-6611Yifeng Sun3Qing Hong4https://orcid.org/0000-0002-6586-7719School of Mechanical Engineering, Nanjing University of Science and Technology, Nanjing, 210094, ChinaSchool of Mechanical Engineering, Nanjing University of Science and Technology, Nanjing, 210094, ChinaSchool of Mechanical Engineering, Southeast University, Nanjing, 210096, ChinaChina Electronics Technology Group 28th Research Institute, Nanjing, 210007, ChinaSchool of Mechanical Engineering, Nanjing University of Science and Technology, Nanjing, 210094, ChinaBackground: Intelligent monitoring of human action in production is an important step to help standardize production processes and construct a digital twin shop-floor rapidly. Human action has a significant impact on the production safety and efficiency of a shop-floor, however, because of the high individual initiative of humans, it is difficult to realize real-time action detection in a digital twin shop-floor. Methods: We proposed a real-time detection approach for shop-floor production action. This approach used the sequence data of continuous human skeleton joints sequences as the input. We then reconstructed the Joint Classification-Regression Recurrent Neural Networks (JCR-RNN) based on Temporal Convolution Network (TCN) and Graph Convolution Network (GCN). We called this approach the Temporal Action Detection Net (TAD-Net), which realized real-time shop-floor production action detection. Results: The results of the verification experiment showed that our approach has achieved a high temporal positioning score, recognition speed, and accuracy when applied to the existing Online Action Detection (OAD) dataset and the Nanjing University of Science and Technology 3 Dimensions (NJUST3D) dataset. TAD-Net can meet the actual needs of the digital twin shop-floor. Conclusions: Our method has higher recognition accuracy, temporal positioning accuracy, and faster running speed than other mainstream network models, it can better meet actual application requirements, and has important research value and practical significance for standardizing shop-floor production processes, reducing production security risks, and contributing to the understanding of real-time production action.https://digitaltwin1.org/articles/1-10/v1Digital twin shop-floor Production action Real-time action detection TAD-Net TCN GCNeng |
| spellingShingle | Liang Fu Yunfeng Xie Tingyu Liu Yifeng Sun Qing Hong TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor [version 1; peer review: 2 approved] Digital Twin Digital twin shop-floor Production action Real-time action detection TAD-Net TCN GCN eng |
| title | TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor [version 1; peer review: 2 approved] |
| title_full | TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor [version 1; peer review: 2 approved] |
| title_fullStr | TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor [version 1; peer review: 2 approved] |
| title_full_unstemmed | TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor [version 1; peer review: 2 approved] |
| title_short | TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor [version 1; peer review: 2 approved] |
| title_sort | tad net an approach for real time action detection based on temporal convolution network and graph convolution network in digital twin shop floor version 1 peer review 2 approved |
| topic | Digital twin shop-floor Production action Real-time action detection TAD-Net TCN GCN eng |
| url | https://digitaltwin1.org/articles/1-10/v1 |
| work_keys_str_mv | AT liangfu tadnetanapproachforrealtimeactiondetectionbasedontemporalconvolutionnetworkandgraphconvolutionnetworkindigitaltwinshopfloorversion1peerreview2approved AT yunfengxie tadnetanapproachforrealtimeactiondetectionbasedontemporalconvolutionnetworkandgraphconvolutionnetworkindigitaltwinshopfloorversion1peerreview2approved AT tingyuliu tadnetanapproachforrealtimeactiondetectionbasedontemporalconvolutionnetworkandgraphconvolutionnetworkindigitaltwinshopfloorversion1peerreview2approved AT yifengsun tadnetanapproachforrealtimeactiondetectionbasedontemporalconvolutionnetworkandgraphconvolutionnetworkindigitaltwinshopfloorversion1peerreview2approved AT qinghong tadnetanapproachforrealtimeactiondetectionbasedontemporalconvolutionnetworkandgraphconvolutionnetworkindigitaltwinshopfloorversion1peerreview2approved |